<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "https://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/xhtml;charset=UTF-8"/>
<meta http-equiv="X-UA-Compatible" content="IE=11"/>
<meta name="generator" content="Doxygen 1.9.4"/>
<meta name="viewport" content="width=device-width, initial-scale=1"/>
<title>Flow: Todo List</title>
<link href="tabs.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="jquery.js"></script>
<script type="text/javascript" src="dynsections.js"></script>
<link href="search/search.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="search/searchdata.js"></script>
<script type="text/javascript" src="search/search.js"></script>
<link href="doxygen.css" rel="stylesheet" type="text/css" />
</head>
<body>
<div id="top"><!-- do not remove this div, it is closed by doxygen! -->
<div id="titlearea">
<table cellspacing="0" cellpadding="0">
 <tbody>
 <tr id="projectrow">
  <td id="projectalign">
   <div id="projectname">Flow<span id="projectnumber">&#160;2.0.0</span>
   </div>
   <div id="projectbrief">Flow project: Full implementation reference.</div>
  </td>
 </tr>
 </tbody>
</table>
</div>
<!-- end header part -->
<!-- Generated by Doxygen 1.9.4 -->
<script type="text/javascript">
/* @license magnet:?xt=urn:btih:d3d9a9a6595521f9666a5e94cc830dab83b65699&amp;dn=expat.txt MIT */
var searchBox = new SearchBox("searchBox", "search",'Search','.html');
/* @license-end */
</script>
<script type="text/javascript" src="menudata.js"></script>
<script type="text/javascript" src="menu.js"></script>
<script type="text/javascript">
/* @license magnet:?xt=urn:btih:d3d9a9a6595521f9666a5e94cc830dab83b65699&amp;dn=expat.txt MIT */
$(function() {
  initMenu('',true,false,'search.php','Search');
  $(document).ready(function() { init_search(); });
});
/* @license-end */
</script>
<div id="main-nav"></div>
<!-- window showing the filter options -->
<div id="MSearchSelectWindow"
     onmouseover="return searchBox.OnSearchSelectShow()"
     onmouseout="return searchBox.OnSearchSelectHide()"
     onkeydown="return searchBox.OnSearchSelectKey(event)">
</div>

<!-- iframe showing the search results (closed by default) -->
<div id="MSearchResultsWindow">
<iframe src="javascript:void(0)" frameborder="0" 
        name="MSearchResults" id="MSearchResults">
</iframe>
</div>

</div><!-- top -->
<div><div class="header">
  <div class="headertitle"><div class="title">Todo List </div></div>
</div><!--header-->
<div class="contents">
<div class="textblock"><dl class="reflist">
<dt>File <a class="el" href="doc-coding__style_8cpp.html">doc-coding_style.cpp</a>  </dt>
<dd><p class="startdd"><a class="anchor" id="_todo000032"></a>In all of Flow, scan for the unintentional use of the period character in auto-brief summaries in many doc headers. A period will be assumed to be the end of the brief summary. We've been disciplined about this, but I (ygoldfel) just realized that, e.g., "boost.asio" has a period and yet is present in at least some doc headers. Fix those in some consistent yet palatable way, so that the Doxygen output for those isn't prematurely truncated. (Check that Doxygen isn't smart enough to figure it out already. If it is, delete this to-do.) </p>
<p class="interdd"><a class="anchor" id="_todo000031"></a>A script/tool/something that would auto-scan the code for violations of cosmetic conventions described in <a class="el" href="doc-coding__style_8cpp.html" title="Doc file (meant for reading, not compiling) that formally defines the coding style for the rest of th...">doc-coding_style.cpp</a>. It need not be perfect or comprehensive. Humans tend to have difficulty following lots of conventions, and it then clutters up code reviews subsequently.</p>
<p class="enddd"><a class="anchor" id="_todo000030"></a>There are a few little to-dos inside the Doxygen-skipped main body of <a class="el" href="doc-coding__style_8cpp.html" title="Doc file (meant for reading, not compiling) that formally defines the coding style for the rest of th...">doc-coding_style.cpp</a>; check them out. </p>
</dd>
<dt>Namespace <a class="el" href="namespaceflow.html">flow</a>  </dt>
<dd><p class="startdd"><a class="anchor" id="_todo000027"></a><code>std::string_view</code> is a way to pass things around similarly to <code>const std::string&amp;</code> without requiring that a <code>string</code> be created just for that purpose; it has a highly similar API but can be constructed from any character sequence in memory and internally stores nothing more than a pointer and length; we should use it wherever possible (within reason) instead of <code>const std::string&amp;</code>. Much code now uses <code>String_view</code>; the remaining to-do is: scour the rest of the code for possible <code>const string&amp;</code>s to convert and indeed convert those to <a class="el" href="namespaceflow_1_1util.html#ae4dc52186797094b3040379ff19124c9" title="Commonly used char-based Basic_string_view. See its doc header.">util::String_view</a>.</p>
<p class="interdd"><a class="anchor" id="_todo000023"></a>One space after period, not two: For some reason in this project I've been following the convention &ndash; in comments and (I think) log messages &ndash; of two spaces after a sentence-ending punctuator (usually period) before the next sentence's first character. I now regret trying this annoying convention. Go back to one space. (For the record, it does look sort of nice, but that's a matter of opinion, and single space looks fine too... AND doesn't confuse various editors' auto-formatting facilityies, among other problem.)</p>
<p class="interdd"><a class="anchor" id="_todo000022"></a>Inline things: Or just use <code>gcc -O3</code> (and non-<code>gcc</code> equivalents) for prettier/faster-to-compile code? The latter is definitely tempting if it works sufficiently well. So far we've been using <code>gcc -O3</code> and equivalents, and it seems to be working well (turning off inlining results in huge performance losses). Still, I am not sure if it would be better to explicitly <code>inline</code> functions instead. Not having to do so definitely simplifies the code, so it is my great hope that the answer is no, and we can keep using <code>gcc -O3</code> and equivalents. In that case delete this paragraph. (To be clear: <code>gcc -O3</code> means that it ignores <code>inline</code> keyword and anything similar, including inlined method bodies inside <code>class {}</code> and <code>struct {}</code>. Instead it determines what to inline based on its own ideas on what will generate the fastest code (with reasonable code size). <code>gcc -O2</code>, on the other hand, will mostly inline things explicitly declared as such in code (again, via <code>inline</code> or inlining inside class bodies or other techniques).) Update: Now using clang (not gcc) with maximum auto-inlining AND FLTO (link-time optimization will allow inlining across object file boundaries) in at least some platforms. This should be close to as good as possible. Update: gcc auto-inlining+FLTO also works.</p>
<p class="interdd"><a class="anchor" id="_todo000021"></a>There are some boost.thread "interruption points" throughout the code, so we should investigate whether we must catch <code>boost::thread_interrupted</code> in those spots, or what...?</p>
<p class="interdd"><a class="anchor" id="_todo000020"></a>Be more specific (cite date) when writing "as of this writing." I use a rhetorical trick when commenting the state of something that may not continue to be the case. Though I do avoid writing such things, often it is necessary; in that case I usually write "as of this writing" or something very similarly worded. That's fine and essentially the best one can do. It means technically the statement won't become false, even if the "sub-statement" (the thing that was true when written) does become false. However, obviously, to the reader of the comment at that later time, that's little consolation: they're still reading a possibly false statement and will want to know what the situation is THEN, or "as of the reading," to to speak. In order to at least try to be helpful, in those cases a date (numeric month/year &ndash; like 4/2017 &ndash; should be sufficient in most cases) should be supplied. The to-do is to convert all "as of this writing" instances &ndash; and to always add a date when writing new instances of "as of this writing." The to-do can be removed once the conversion is completed. Example: this to-do has not been completed as of this writing (11/2017). (Side note: possibly goes without saying, but one is free to explain to an arbitrary degree of detail why something is true as of that writing, and how/why/when it might change. This to-do covers those cases where no such explanation is written. It would be impractically verbose to get into speculative detail for every as-of-this-writing instance; so at least a date should thus be inserted.)</p>
<p class="interdd"><a class="anchor" id="_todo000019"></a>The comments (as of this writing, all written by me, ygoldfel) in this library could use an edit to make them briefer. (I've found even a self-edit by me, with that express purpose, often does wonders.) Background: I write very extensive comments. I am quite convinced this is superior (far superior even) to next-to-no comments; and to the average amount of comments one tends to see in production code. <em>That said</em>, the top code review feedback I receive (not here specifically but in general) is that my comments tend to be too "discursive" (consisting of discourse) and/or at times unnecessarily in-depth. Discursive = as if I am talking to the reader (many prefer a terser, more formal style). Too in-depth = tends to go into history, related issues, future work, etc. (these elements can remain but can be cut down significant without losing much substance). In other words, there should be a happy middle in terms of comment volume, and this can be achieved by a comment edit run by Yuri or someone else (if reviewed by Yuri). To be clear, I believe this middle ground is to be closer to the status quo than to the average amount of comments in other projects.</p>
<p class="interdd"><a class="anchor" id="_todo000018"></a>Possibly document exceptions thrown explicitly via the Doxygen keyword meant for this purpose: <code>"@throws"</code>. Currently when we document explicitly throwing an exception, it is ALWAYS a <a class="el" href="classflow_1_1error_1_1Runtime__error.html" title="An std::runtime_error (which is an std::exception) that stores an Error_code.">flow::error::Runtime_error</a> encapsulating a <a class="el" href="namespaceflow.html#a29eaaa9d0fac4ce87d8b969222dbed09" title="Short-hand for a boost.system error code (which basically encapsulates an integer/enum error code and...">flow::Error_code</a> (which is an <code>int</code>-like error code). This is very explicitly documented, but technically Doxygen has a keyword which will generate a special little readout for the exception (similarly as for each parameter, etc.). We don't use that keyword. We probably should, though this isn't totally cut-and-dried. Consider that we already document the exception on the <code>err_code</code> parameter in every case; so no information would really be gained (only arguably nicer formatting). On the other hand, the code would be somewhat more verbose (and boiler-platey, since each already boiler-platey <code>err_code</code> comment snippet would essentially grow in size). Furthermore, if we document this explicit exception, one might say it behooves us to now document all the other possible sources of exceptions such as <code>std::bad_alloc</code> when running out of heap memory. Perhaps then we have to talk about constructor-throwing-exception behavior and other C++ technicalities to do with exceptions. Do we really want to enter that land? I think maybe not; consider just leaving it alone. Though, maybe I'm over-dramatizing the impact of adding a <code>"@throws"</code> section on our various <a class="el" href="classflow_1_1error_1_1Runtime__error.html" title="An std::runtime_error (which is an std::exception) that stores an Error_code.">flow::error::Runtime_error</a>-throwing methods. Well, it's a to-do; decide later.</p>
<p class="interdd"><a class="anchor" id="_todo000017"></a>Since Flow gained its first users beyond the original author, some Flow-adjacent code has been written from which Flow can benefit, including a potential <code>io</code> module/namespace for general networking/local I/O. (Flow itself continued to be developed, but some features were added elsewhere for expediency; this is a reminder to factor them out into Flow for the benefit of all.) Some features to migrate here might be: boost.asio extensions to UDP receive APIs to obtain receipt time stamps and destination IP (<code>recvmsg()</code> with ancillary data extensions) and to receive multiple datagrams in one call (<code>recvmmsg()</code>); boost.asio-adjacent facility to add custom socket options similarly to how boost.asio does it internally; boost.asio support for (local) transmission of native socket handles and security data over stream sockets (<code>SCM_RIGHTS</code>, etc.).</p>
<p class="interdd"><a class="anchor" id="_todo000016"></a>As of this writing the <em>exact</em> nature of where the project will permanently reside (and who will maintain it vs. use it) is in flux. Therefore for now I have removed the section covering certain topics and replaced it with the to-do you're reading. This should be undone when things settle down (obviously ensuring the brought-back section is made accurate). The topics to cover: <code>"@author"</code> (including contact info); GitHub/other address indicating where to browse the project source; link(s) to the latest auto-generated web docs (if any); a section on the history of the project; and licensing info (if deemed needed) or pointer to it. (Reminder: Also update any similar sections of the historically significant <a class="el" href="classflow_1_1net__flow_1_1Node.html" title="An object of this class is a single Flow-protocol networking node, in the sense that: (1) it has a di...">net_flow::Node</a> doc header.)</p>
<p class="interdd"><a class="anchor" id="_todo000024"></a>We use <code>0</code> instead of <code>NULL</code> or <code>nullptr</code> when needing a null pointer; perhaps we should use the latter. <code>NULL</code> is an anachronism from C, so we shouldn't use it. <code>nullptr</code> is at least no worse than <code>0</code>, however, other than being less concise. However, the main reason it exists &ndash; to avoid ambiguities in function overloading (e.g., when something could take either an <code>int</code> or a <code>char*</code>, <code>nullptr</code> would resolve to the latter, while <code>0</code> probably unintentionally to the former) &ndash; is not a situation our style ever invokes, to my knowledge, so using <code>nullptr</code> would not solve any actual problems. However, it could be argued that using it more readily calls attention to the use of a pointer, as opposed to an integer, in the particular context at play. So it's something to consider (but, no doubt, the conversion process would be laborious, as there's no simple search-replace that would work).</p>
<p class="interdd"><a class="anchor" id="_todo000025"></a><code>= default</code> for copy constructors and copy operators is now used in a few places; consider spreading this C++11 feature everywhere it's being done implicitly due to C++03 rules (good documentation practices suggest declaring them explicitly but of course leave the implementation to the compiler default, gaining best of both worlds &ndash; proper class API docs yet maintenance-friendly default body).</p>
<p class="interdd"><a class="anchor" id="_todo000026"></a>Consider PIMPL and related topics. Recommend scouring Boost docs, particularly for the smart pointer library, which discuss how to potentially use smart pointers for easy PIMPLing. In general, research the state of the art on the topic of library interface vs. implementation/hiding.</p>
<p class="interdd"><a class="anchor" id="_todo000028"></a>Return-by-copy binary operators of the form <code>T operatorBLAH(const T&amp; x1, const Some_type&amp; x2)</code> should be written as free functions instead of <code>T</code> members. I don't recall at this point why, but this tends to be recommended and done in STL and Boost. Maybe check the Meyer Effective C++ book on the theory; and if it makes sense find all such operators written as members and change them to be free functions. Should this be avoided if it requires <code>friend</code> though? Lastly, for Doxygen, use the <code>relatesalso T</code> command to link the free function to the class <code>T</code> in the documentation.</p>
<p class="enddd"><a class="anchor" id="_todo000029"></a>In many (most) cases we pass <code>shared_ptr</code>s (and their aliases) by value when it would be more performant to do so by <code>const</code> reference; at times possibly better to pass by raw pointer. Scott Meyer/Herb Sutter have opined on this to basically indicate that (1) it is often best to use a raw pointer, unless actual copying/ownership status is expected; but failing that (2) it is often best to use a <code>const&amp;</code> when safe; and failing that passing by value is fine. This need not be a dogmatic change, but we should be more mindful than simply always passing by value. When searching for instances to potentially change, check for <code>shared_ptr</code>, <code>Ptr</code>, and <code>_ptr</code> tokens.  </p>
</dd>
<dt>Namespace <a class="el" href="namespaceflow_1_1async.html">flow::async</a>  </dt>
<dd><a class="anchor" id="_todo000001"></a>The thread-to-core optimizations provided at this time are, at least, a good start, but more advanced logic can be devised with more low-level experience and/or by using certain open-source libraries. It's possible that a more knowledgeable person would devise more or better knobs and/or require less manual specification of values. The following background reading may help devise more advanced logic and/or knobs: [ <a href="https://eli.thegreenplace.net/2016/c11-threads-affinity-and-hyperthreading/">https://eli.thegreenplace.net/2016/c11-threads-affinity-and-hyperthreading/</a> | <a href="https://mirrors.edge.kernel.org/pub/linux/kernel/people/paulmck/perfbook/perfbook.2016.07.31a.pdf">https://mirrors.edge.kernel.org/pub/linux/kernel/people/paulmck/perfbook/perfbook.2016.07.31a.pdf</a> | <a href="https://lwn.net/Articles/255364/">https://lwn.net/Articles/255364/</a> | "hwloc" library (portable lib for detailed hardware topology info) | libNUMA ].  </dd>
<dt>Member <a class="el" href="classflow_1_1async_1_1Concurrent__task__loop.html#ac6b98b440d52c079c22d7972c9b12e1b">flow::async::Concurrent_task_loop::schedule_from_now</a>  (const Fine_duration &amp;from_now, Scheduled_task &amp;&amp;task)=0</dt>
<dd><a class="anchor" id="_todo000005"></a>Deal with the scheduled-tasks-affected-from-outside-loop corner case of the <code>Concurrent_task_loop::schedule_*()</code> APIs. Perhaps add <code>bool in_loop_use_only</code> arg which, if <code>false</code>, will always disable the <code>single_threaded</code> optimization internally. At this time it always enables it if <code><a class="el" href="classflow_1_1async_1_1Concurrent__task__loop.html#a131335863b19446dbc349de181d4b5a1" title="How many threads does start() start?">n_threads()</a> == 1</code> which will cause thread un-safety if the returned handle is touched from outside an in-loop task. <code>void</code> versions of the <code>schedule_*()</code> APIs should be added which would lack this, as in that case there is no handle to misuse outside the loop. </dd>
<dt>Member <a class="el" href="classflow_1_1async_1_1Concurrent__task__loop.html#afff5e62840791b309e178c5ddc26e594">flow::async::Concurrent_task_loop::start</a>  (Task &amp;&amp;init_task_or_empty=<a class="el" href="namespaceflow_1_1async.html#af35fb65bdea1a3a92929ec2c4a6a6b72" title="Short-hand for a task that can be posted for execution by a Concurrent_task_loop or flow::util::Task_...">Task()</a>, const Thread_init_func &amp;thread_init_func_or_empty=<a class="el" href="classflow_1_1async_1_1Concurrent__task__loop.html#aa164fb1fe1b3d7fce0de0174d77ea396" title="Short-hand for the thread-initializer-function optional arg type to start().">Thread_init_func()</a>)=0</dt>
<dd><a class="anchor" id="_todo000003"></a><a class="el" href="classflow_1_1async_1_1Concurrent__task__loop.html#afff5e62840791b309e178c5ddc26e594" title="Starts all threads in the thread pool; any queued post()ed (and similar) tasks may begin executing im...">Concurrent_task_loop::start()</a> has an optional thread-initializer-function arg; it could be reasonable to ass a thread-finalizer-function arg symmetrically. As of this writing there is no use case, but it's certainly conceivable. </dd>
<dt>Member <a class="el" href="classflow_1_1async_1_1Concurrent__task__loop.html#aadadb5b8b6c40ddc86c205de1b8f25b1">flow::async::Concurrent_task_loop::stop</a>  ()=0</dt>
<dd><a class="anchor" id="_todo000004"></a>boost.asio has advanced features that might help to mark certain tasks as "must-run if already queued, even
if one `stop()`s"; consider providing user-friendly access to these features, perhaps in the context of the existing <a class="el" href="classflow_1_1async_1_1Concurrent__task__loop.html#aadadb5b8b6c40ddc86c205de1b8f25b1" title="Waits for any ongoing task(s)/completion handler(s) to return; then prevents any further-queued such ...">Concurrent_task_loop::stop()</a> API. These features are documented as of this writing at least in the <code>io_context</code> doc page.  </dd>
<dt>Class <a class="el" href="classflow_1_1async_1_1Cross__thread__task__loop.html">flow::async::Cross_thread_task_loop</a>  </dt>
<dd><a class="anchor" id="_todo000009"></a>Add dynamic configurability of low-level thread/core behavior of <a class="el" href="classflow_1_1async_1_1Cross__thread__task__loop.html" title="Concrete Concurrent_task_loop that is able to efficiently schedule Tasks within a given Op to execute...">Cross_thread_task_loop</a>, <a class="el" href="classflow_1_1async_1_1Segregated__thread__task__loop.html" title="Concrete Concurrent_task_loop that uses the legacy pin-Tasks-within-1-Op-to-1-thread method of achiev...">Segregated_thread_task_loop</a>, and the <a class="el" href="classflow_1_1async_1_1Concurrent__task__loop.html" title="The core flow::async interface, providing an optionally multi-threaded thread pool onto which runnabl...">Concurrent_task_loop</a> interface generally, as these parameters (including <code>n_threads_or_zero</code>, <code>est_hw_core_sharing_helps_algo</code>, <code>est_hw_core_pinning_helps_algo</code>) can only be set at construction time even though <a class="el" href="classflow_1_1async_1_1Cross__thread__task__loop.html#a5705e34163dec8463a30361bdb2fa83f" title="Implements superclass API.">start()</a> and <a class="el" href="classflow_1_1async_1_1Cross__thread__task__loop.html#aa0f53f269524b701b23237766f79df2d" title="Implements superclass API.">stop()</a> can be invoked anytime. For instance a non-<code>virtual</code> <code>configure()</code> method could be added to each <a class="el" href="classflow_1_1async_1_1Concurrent__task__loop.html" title="The core flow::async interface, providing an optionally multi-threaded thread pool onto which runnabl...">Concurrent_task_loop</a> subclass, potentially with different signatures. Note, also, that the decision to have <a class="el" href="classflow_1_1async_1_1Cross__thread__task__loop.html#aa0f53f269524b701b23237766f79df2d" title="Implements superclass API.">stop()</a> exit the actual threads, in addition to making some things easier to code/reason about internally, is also more amenable to these dynamic changes &ndash; messing with threading behavior dynamically is easier if one can exit and re-spawn the thread pool. </dd>
<dt>Member <a class="el" href="namespaceflow_1_1async.html#a0a06794b16b72f2829ce3353557c8485a57525f29ac96f4edd56c951e7d7c7b25">flow::async::S_ASYNC_AND_AWAIT_CONCURRENT_COMPLETION</a>  </dt>
<dd><a class="anchor" id="_todo000002"></a>Much like the promise/future mechanism provides optional timed wait functionality, it might make sense to provide the API ability to set an optional time limit for any wait invoked by <a class="el" href="namespaceflow_1_1async.html#a0a06794b16b72f2829ce3353557c8485a57525f29ac96f4edd56c951e7d7c7b25" title="Same as Synchronicity::S_ASYNC but the posting routine then waits as long as necessary for the given ...">Synchronicity::S_ASYNC_AND_AWAIT_CONCURRENT_COMPLETION</a> or <a class="el" href="namespaceflow_1_1async.html#a0a06794b16b72f2829ce3353557c8485ae9a9ae292ea3fbbe3b313348dcec0846" title="Same as Synchronicity::S_ASYNC but the posting routine then waits as long as necessary for the given ...">Synchronicity::S_ASYNC_AND_AWAIT_CONCURRENT_START</a>. Probably best to add this only once a need clearly arises though.  </dd>
<dt>Class <a class="el" href="classflow_1_1async_1_1Segregated__thread__task__loop.html">flow::async::Segregated_thread_task_loop</a>  </dt>
<dd><p class="startdd"><a class="anchor" id="_todo000007"></a>In <a class="el" href="classflow_1_1async_1_1Segregated__thread__task__loop.html" title="Concrete Concurrent_task_loop that uses the legacy pin-Tasks-within-1-Op-to-1-thread method of achiev...">Segregated_thread_task_loop</a>, when randomly selecting a thread, considering trying to bias in favor of threads with less load (perhaps measured as # of tasks enqueued in a given thread's <code>Task_engine</code>); and/or round-robin thread assignment; and/or other heuristics. </p>
<p class="enddd"><a class="anchor" id="_todo000006"></a>In <a class="el" href="classflow_1_1async_1_1Segregated__thread__task__loop.html" title="Concrete Concurrent_task_loop that uses the legacy pin-Tasks-within-1-Op-to-1-thread method of achiev...">Segregated_thread_task_loop</a>, when dealing with the 2-arg <code><a class="el" href="classflow_1_1async_1_1Segregated__thread__task__loop.html#a9ae5be313170da86f361f8bf987b40b4" title="Implements superclass API.">post()</a></code> (the one taking an <a class="el" href="namespaceflow_1_1async.html#a024042c64b7dc544a7a5587afa4b1949" title="An object of this opaque type represents a collection of 1 or more async::Task, past or future,...">async::Op</a> and an <a class="el" href="namespaceflow_1_1async.html#af35fb65bdea1a3a92929ec2c4a6a6b72" title="Short-hand for a task that can be posted for execution by a Concurrent_task_loop or flow::util::Task_...">async::Task</a>) and similar <code>schedule_*()</code> methods &ndash; consider postponing the random thread selection until the last possible moment, as opposed to in <a class="el" href="classflow_1_1async_1_1Segregated__thread__task__loop.html#a8257d8aa45ea527ce6c0783e37346743" title="Implements superclass API.">create_op()</a>. Given that <a class="el" href="classflow_1_1async_1_1Segregated__thread__task__loop.html" title="Concrete Concurrent_task_loop that uses the legacy pin-Tasks-within-1-Op-to-1-thread method of achiev...">Segregated_thread_task_loop</a> is legacy-ish in nature, the resulting significant increase in internal complexity may or may not be worth accepting. This to-do also makes little practical difference without the nearby to-do that would bias in favor of less-loaded threads. </p>
</dd>
<dt>Member <a class="el" href="classflow_1_1async_1_1Single__thread__task__loop.html#a2669021fe545a05ced370f09ce794f52">flow::async::Single_thread_task_loop::schedule_from_now</a>  (const Fine_duration &amp;from_now, Scheduled_task &amp;&amp;task)</dt>
<dd><a class="anchor" id="_todo000008"></a>Deal with the scheduled-tasks-affected-from-outside-loop corner case of the <code>Single_thread_task_loop::schedule_*()</code> APIs. Perhaps add <code>bool in_loop_use_only</code> arg which, if <code>false</code>, will always disable the <code>single_threaded</code> optimization internally. At this time it always enables it which will cause thread un-safety if the returned handle is touched from outside an in-loop task. <code>void</code> versions of the <code>schedule_*()</code> APIs should be added which would lack this, as in that case there is no handle to misuse outside the loop. </dd>
<dt>Class <a class="el" href="classflow_1_1cfg_1_1Config__manager.html">flow::cfg::Config_manager&lt; S_d_value_set &gt;</a>  </dt>
<dd><a class="anchor" id="_todo000010"></a>Dynamic config support lacks crash rejection; though these features should be an incremental improvement around the existing code. By crash rejection I mean something like: config file X comes in; we rename it to X.unparsed; we try to parse it; program crashes &ndash; versus it's fine, so we rename it X.parsed, and next time X.parsed is what we presumably-safely parse after any restart. Similarly invalid config may not cause a crash but still shouldn't be repeatedly re-parsed (etc.). Exact design TBD and will be informed by previous work on other projects (by echan and ygoldfel at least). </dd>
<dt>Member <a class="el" href="classflow_1_1cfg_1_1Config__manager.html#a22bfec8082d0c7b5dff9ee7e98865707">flow::cfg::Config_manager&lt; S_d_value_set &gt;::apply_static</a>  (const fs::path &amp;static_cfg_path, const typename <a class="el" href="structflow_1_1cfg_1_1Final__validator__func.html#a5475ab0b6e93f2ccded2aa8690ace0a4" title="Short-hand for a function that takes a parsed config set (meaning all values are individually OK) and...">Final_validator_func&lt; Value_set &gt;::Type</a> &amp;... final_validator_func, bool commit=true)</dt>
<dd><a class="anchor" id="_todo000011"></a>Add support for command line as a config source in addition to file(s), for static config in <a class="el" href="classflow_1_1cfg_1_1Config__manager.html" title="Manages a config setup, intended for a single daemon process, by maintaining 1 or more set(s) of stat...">cfg::Config_manager</a> and <a class="el" href="classflow_1_1cfg_1_1Static__config__manager.html" title="A Config_manager-related adapter-style class that manages a simple config setup involving a single (t...">cfg::Static_config_manager</a>. </dd>
<dt>Class <a class="el" href="classflow_1_1cfg_1_1Dynamic__cfg__context.html">flow::cfg::Dynamic_cfg_context&lt; Root, Target, Target_ptr &gt;</a>  </dt>
<dd><a class="anchor" id="_todo000012"></a><a class="el" href="classflow_1_1cfg_1_1Dynamic__cfg__context.html" title="Class which facilitates managing access to a dynamic configuration.">flow::cfg::Dynamic_cfg_context</a> <code>Target_ptr</code> is deprecated and shall be always left at its default value in future code; eventually remove it entirely and hard-code the default value internally.  </dd>
<dt>Class <a class="el" href="classflow_1_1cfg_1_1Option__set.html">flow::cfg::Option_set&lt; Value_set &gt;</a>  </dt>
<dd><a class="anchor" id="_todo000013"></a>Add individual-option-changed detection API(s) in addition to the existing <a class="el" href="classflow_1_1cfg_1_1Option__set.html" title="The core config-parsing facility, which builds parsing/comparison/output capabilities on top of a giv...">Option_set</a> overall-value-set-changed detection API. This is contingent on a use case needing it. The existing code already detects this internally and logs a message for each changed option (in <a class="el" href="classflow_1_1cfg_1_1Option__set.html#aa1f780bfd3a5ff07ade1e36785e28f49" title="In PARSING state enters CANONICAL state, finalizing values() from values_candidate().">canonicalize_candidate()</a>). </dd>
<dt>Member <a class="el" href="structflow_1_1cfg_1_1Option__set_1_1Declare__options__func__args.html#ae0e0d11ce1bf4ce971f5e2d75ed6e18d">flow::cfg::Option_set&lt; Value_set &gt;::Declare_options_func_args::m_args</a>  </dt>
<dd><a class="anchor" id="_todo000014"></a><code>union</code> used inside <a class="el" href="classflow_1_1cfg_1_1Option__set.html" title="The core config-parsing facility, which builds parsing/comparison/output capabilities on top of a giv...">Option_set</a> implementation should be replaced with the type-safe, equally-fast, more-expressive <code>std::variant</code>. The author was not aware of the latter's existence when originally writing this.  </dd>
<dt>Member <a class="el" href="classflow_1_1cfg_1_1Static__config__manager.html#aacacd2c0eddaf68d0dc5e0c1407572f5">flow::cfg::Static_config_manager&lt; Value_set &gt;::apply</a>  (const fs::path &amp;cfg_path, const typename <a class="el" href="structflow_1_1cfg_1_1Final__validator__func.html#a5475ab0b6e93f2ccded2aa8690ace0a4" title="Short-hand for a function that takes a parsed config set (meaning all values are individually OK) and...">Final_validator_func&lt; Value_set &gt;::Type</a> &amp;final_validator_func)</dt>
<dd><a class="anchor" id="_todo000015"></a>Add support for command line as a config source in addition to file(s), for static config in <a class="el" href="classflow_1_1cfg_1_1Config__manager.html" title="Manages a config setup, intended for a single daemon process, by maintaining 1 or more set(s) of stat...">cfg::Config_manager</a>. </dd>
<dt>Namespace <a class="el" href="namespaceflow_1_1log.html">flow::log</a>  </dt>
<dd><p class="startdd"><a class="anchor" id="_todo000055"></a>Lacking feature: log message rate-limiting. This could actually mean a few things. One is being able to rate-limit the messages produced a given log call site per unit time; this might be the minimum for this to-do. This could be done by a dirty macro hack; or it could be done in more civilized fashion (while minding perf) by configuring it by message ID (but message IDs are optional and not implemented as of this writing anyway). Another rate-limiting thing is general controls on frequency of logging; though arguably that is more applicable to individual <code><a class="el" href="classflow_1_1log_1_1Logger.html" title="Interface that the user should implement, passing the implementing Logger into logging classes (Flow&#39;...">Logger</a></code> implementations rather than as a mandatory-general mechanism.</p>
<p class="interdd"><a class="anchor" id="_todo000054"></a>Lacking feature: boost.format-style logging call sites. This is conceptually similar to &ndash; and possibly even entirely subsuming in backwards-compatible fashion &ndash; the to-do just above for <code>printf()</code>-style logging. Likely both to-dos should be designed/implemented in one shot. Moreover, it is possible (and would be absolutely delightful if true) that boost.format's <code>format</code> class can be used entirely transparently on top of the <code>ostream</code>-focused existing <a class="el" href="namespaceflow_1_1log.html" title="Flow module providing logging functionality.">flow::log</a> API (such as <a class="el" href="log_8hpp.html#a626c7dc4d3b4dc0b32a8aac8624d66bc" title="Logs a WARNING message into flow::log::Logger *get_logger() with flow::log::Component get_log_compone...">FLOW_LOG_WARNING()</a> and buddies)!</p>
<p class="interdd"><a class="anchor" id="_todo000053"></a>Lacking feature: <code>printf()</code>-style logging call sites, in contrast to the currently supported <code>ostream</code> fragment call sites. So, like, <code>FLOW_LOG_WARNING_FMT("Hello, Mr. %s!", my_name.c_str())</code> could be used analogously to <code>FLOW_LOG_WARNING("Hello, Mr " &lt;&lt; my_name &lt;&lt; "!!")</code> (with <code>std::string my_name</code>). In terms of implementation, there is more to this than meets the eye perhaps; as <code>ostream</code>s potentially store (formatting) state between totally separate invocations of <code>ostream&lt;&lt;</code>, whereas <code>printf()</code> style functions normally do not. Internally, this likely allows for specially optimized logging code.</p>
<p class="interdd"><a class="anchor" id="_todo000052"></a>Lacking feature: message IDs. This is discussed a bit more in the <a class="el" href="structflow_1_1log_1_1Msg__metadata.html" title="Simple data store containing all of the information generated at every logging call site by flow::log...">log::Msg_metadata</a> doc header.</p>
<p class="interdd"><a class="anchor" id="_todo000051"></a>There are third-party logging systems including boost.log. The to-do is to investigate using boost.log, partially or entirely replacing the manually implemented system. (To be honest I am rather partial to the simple but effective system already implemented, in particular the performance-focused trickery.) Boost is invaluable in many ways, and Flow modules use it extremely extensively, but my intuition says that in this case it may be irrelevant. Meanwhile, boost.log can, of course, be used within user's <a class="el" href="classflow_1_1log_1_1Logger.html" title="Interface that the user should implement, passing the implementing Logger into logging classes (Flow&#39;...">Logger</a> interface implementation, but that's not what this to-do is about; it's about actually placing it all over Flow's implementation and APIs <em>in place of</em> <a class="el" href="classflow_1_1log_1_1Logger.html" title="Interface that the user should implement, passing the implementing Logger into logging classes (Flow&#39;...">Logger</a>, <a class="el" href="namespaceflow_1_1log.html#a5c602384e6fe9d7642740ac2fae4ad04" title="Enumeration containing one of several message severity levels, ordered from highest to lowest.">log::Sev</a>, etc. (Also, after a 5-minute glance at boost.log, I am noticing that some of its ideas I seem to have independently and unknowingly replicated here. It is, however, simple stuff, not rocket science.) My intuition is I hesitate to force-marry the Flow user to boost.log; the very simple <a class="el" href="classflow_1_1log_1_1Logger.html" title="Interface that the user should implement, passing the implementing Logger into logging classes (Flow&#39;...">Logger</a> (abstract) interface is seemingly a gentler thing to force the Flow user into, at least relatively speaking.</p>
<p class="enddd"><a class="anchor" id="_todo000056"></a>Lacking feature: compiler hints for optimizing away log filter checks. This is inspired by a certain other proprietary logging API in which we noticed attempts to give hints to the compiler as to how likely or unlikely the verbosity check is to return a true or false value, seemingly so that the compiler might optimize out the check and hence the entire log statement in some conditions; or <em>always</em> log others and skip the check in the optimized code. I omit any details here, but that's the general idea. I don't know how effective this is given that verbosities are configurable dynamically potentially; but look into it.  </p>
</dd>
<dt>Class <a class="el" href="classflow_1_1log_1_1Async__file__logger.html">flow::log::Async_file_logger</a>  </dt>
<dd><a class="anchor" id="_todo000033"></a>Lacking feature: Compress-as-you-log in <a class="el" href="classflow_1_1log_1_1Async__file__logger.html" title="An implementation of Logger that logs messages to a given file-system path but never blocks any loggi...">Async_file_logger</a>. So, optionally, when characters are actually written out to file-system, gzip/zip/whatever them instead of writing plain text. (This is possible at least for gzip.) Background: It is common-place to compress a log file after it has been rotated (e.g., around rotation time: F.log.1.gz -&gt; F.log.2.gz, F.log -&gt; F.log.1 -&gt; F.log.1.gz). It is more space-efficient (at least), however, to write to F.log.gz directly already in compressed form; then rotation requires only renaming (e.g.: F.log.1.gz -&gt; F.log.2.gz, F.log.gz [already gzipped from the start] -&gt; F.log.1.gz). </dd>
<dt>Member <a class="el" href="classflow_1_1log_1_1Async__file__logger.html#aec2b1796c81b020cacea493e21e19562">flow::log::Async_file_logger::Async_file_logger</a>  (<a class="el" href="classflow_1_1log_1_1Logger.html" title="Interface that the user should implement, passing the implementing Logger into logging classes (Flow&#39;...">Logger</a> *backup_logger_ptr, <a class="el" href="classflow_1_1log_1_1Config.html" title="Class used to configure the filtering and logging behavior of Loggers; its use in your custom Loggers...">Config</a> *config, const fs::path &amp;log_path, bool capture_rotate_signals_internally)</dt>
<dd><a class="anchor" id="_todo000034"></a>Consider adding <a class="el" href="classflow_1_1log_1_1Async__file__logger.html" title="An implementation of Logger that logs messages to a given file-system path but never blocks any loggi...">Async_file_logger</a> constructor option to overwrite the file instead of appending. </dd>
<dt>Member <a class="el" href="classflow_1_1log_1_1Async__file__logger.html#aaaf956d63042cbc0800680996adabfa2">flow::log::Async_file_logger::log_flush_and_reopen</a>  (bool async=true)</dt>
<dd><a class="anchor" id="_todo000035"></a><code>Async_file_logger::log_flush_and_reopen(true)</code> is great for flushing, such as in an abort-signal handler, but providing just the flushing part without the reopening might be useful. At the moment we've left it this way, due to the vague feeling that closing the file upon flushing it is somehow more final and thus safer (in terms of accomplishing its goal) in an abort-signal scenario. Feelings aren't very scientific though. </dd>
<dt>Member <a class="el" href="classflow_1_1log_1_1Async__file__logger.html#a5c06c592797c70077fc02c98ebf231da">flow::log::Async_file_logger::m_serial_logger</a>  </dt>
<dd><a class="anchor" id="_todo000036"></a><a class="el" href="classflow_1_1log_1_1Async__file__logger.html#a5c06c592797c70077fc02c98ebf231da" title="This is the Logger doing all the real log-writing work (the one stored in Log_context is the logger-a...">Async_file_logger::m_serial_logger</a> (and likely a few other similar <code>unique_ptr</code> members of other classes) would be slightly nicer as an <code>std::optional&lt;&gt;</code> instead of <code>unique_ptr</code>. <code>optional</code> was not in STL back in the day and either did not exist or did not catch our attention back in the day. <code>unique_ptr</code> in situations like this is fine but uses the heap more.  </dd>
<dt>Class <a class="el" href="classflow_1_1log_1_1Config.html">flow::log::Config</a>  </dt>
<dd><a class="anchor" id="_todo000037"></a>Class <a class="el" href="classflow_1_1log_1_1Config.html" title="Class used to configure the filtering and logging behavior of Loggers; its use in your custom Loggers...">Config</a> doc header is wordy and might be hard to follow; rewrite for clarity/flow/size. </dd>
<dt>Member <a class="el" href="classflow_1_1log_1_1Config.html#ab067e9e513530e7484a30f568b1e11d5">flow::log::Config::Config</a>  (<a class="el" href="classflow_1_1log_1_1Config.html" title="Class used to configure the filtering and logging behavior of Loggers; its use in your custom Loggers...">Config</a> &amp;&amp;)=delete</dt>
<dd><a class="anchor" id="_todo000038"></a>Reconsider providing a <a class="el" href="classflow_1_1log_1_1Config.html" title="Class used to configure the filtering and logging behavior of Loggers; its use in your custom Loggers...">Config</a> move constructor. I just didn't need to deal with it.  </dd>
<dt>Member <a class="el" href="classflow_1_1log_1_1Config.html#ac8a787b8026d84814bf4edcc91432447">flow::log::Config::operator=</a>  (<a class="el" href="classflow_1_1log_1_1Config.html" title="Class used to configure the filtering and logging behavior of Loggers; its use in your custom Loggers...">Config</a> &amp;&amp;)=delete</dt>
<dd><a class="anchor" id="_todo000040"></a>Reconsider providing a <a class="el" href="classflow_1_1log_1_1Config.html" title="Class used to configure the filtering and logging behavior of Loggers; its use in your custom Loggers...">Config</a> move assignment. I just didn't need to deal with it.  </dd>
<dt>Member <a class="el" href="classflow_1_1log_1_1Config.html#a60c9a16ebb764e4fcc4208a55c2e785d">flow::log::Config::operator=</a>  (const <a class="el" href="classflow_1_1log_1_1Config.html" title="Class used to configure the filtering and logging behavior of Loggers; its use in your custom Loggers...">Config</a> &amp;)=delete</dt>
<dd><a class="anchor" id="_todo000039"></a>Reconsider providing a <a class="el" href="classflow_1_1log_1_1Config.html" title="Class used to configure the filtering and logging behavior of Loggers; its use in your custom Loggers...">Config</a> copy assignment. I just didn't need to deal with it.  </dd>
<dt>Member <a class="el" href="classflow_1_1log_1_1Logger.html#a1ea9da10a13d3b422ba7162a13d7eeb1">flow::log::Logger::set_thread_info</a>  (std::string *call_thread_nickname, <a class="el" href="namespaceflow_1_1util.html#a0a4e7b2d3107bd630e0dc0c0b8f2e4de" title="Short-hand for an OS-provided ID of a util::Thread.">flow::util::Thread_id</a> *call_thread_id)</dt>
<dd><a class="anchor" id="_todo000050"></a>It would be more consistent to rename <a class="el" href="classflow_1_1log_1_1Logger.html#a1ea9da10a13d3b422ba7162a13d7eeb1" title="Same as set_thread_info_in_msg_metadata() but targets the given two variables as opposed to a Msg_met...">set_thread_info()</a> to this_thread_set_info(), since it operates in thread-local fashion. This was a naming convention oversight. </dd>
<dt>Member <a class="el" href="classflow_1_1log_1_1Logger.html#a2d1a0283764e5f85abf00c7cee4c4e19">flow::log::Logger::set_thread_info_in_msg_metadata</a>  (<a class="el" href="structflow_1_1log_1_1Msg__metadata.html" title="Simple data store containing all of the information generated at every logging call site by flow::log...">Msg_metadata</a> *msg_metadata)</dt>
<dd><a class="anchor" id="_todo000049"></a>It would be more consistent to rename <a class="el" href="classflow_1_1log_1_1Logger.html#a2d1a0283764e5f85abf00c7cee4c4e19" title="Loads msg_metadata-&gt;m_call_thread_nickname (if set) or else msg_metadata-&gt;m_call_thread_id,...">set_thread_info_in_msg_metadata()</a> to this_thread_set_info_in_msg_metadata(), since it operates in thread-local fashion. This was a naming convention oversight. </dd>
<dt>Member <a class="el" href="classflow_1_1log_1_1Logger.html#a5600c71ccaa294650c7d34b596d90bf5">flow::log::Logger::this_thread_set_logged_nickname</a>  (<a class="el" href="namespaceflow_1_1util.html#ae4dc52186797094b3040379ff19124c9" title="Commonly used char-based Basic_string_view. See its doc header.">util::String_view</a> thread_nickname=<a class="el" href="namespaceflow_1_1util.html#ae4dc52186797094b3040379ff19124c9" title="Commonly used char-based Basic_string_view. See its doc header.">util::String_view()</a>, <a class="el" href="classflow_1_1log_1_1Logger.html" title="Interface that the user should implement, passing the implementing Logger into logging classes (Flow&#39;...">Logger</a> *logger_ptr=0, bool also_set_os_name=true)</dt>
<dd><a class="anchor" id="_todo000048"></a><a class="el" href="classflow_1_1log_1_1Logger.html#a5600c71ccaa294650c7d34b596d90bf5" title="Sets or unsets the current thread&#39;s logging-worthy string name; optionally sets the OS thread name (s...">this_thread_set_logged_nickname()</a> could take multiple <code><a class="el" href="classflow_1_1log_1_1Logger.html" title="Interface that the user should implement, passing the implementing Logger into logging classes (Flow&#39;...">Logger</a></code>s, since it is an app-wide function and potentially would want to be reflected in multiple loggers. It could take a pair of iterators or a container (in both cases, of template argument type). Arguably, though, this is overkill; as (1) most code paths deal with only one <code>get_logger()</code>-owning object each; (2) naming of a thread is obvious enough in subsequent logs (hence this message only calls attention to that operation plus notes the thread ID before the nickname assignment, not necessarily the most critical of information). Certainly this zero-to-one-<code><a class="el" href="classflow_1_1log_1_1Logger.html" title="Interface that the user should implement, passing the implementing Logger into logging classes (Flow&#39;...">Logger</a></code> version must continue to be available for syntactic-sugary convenience, even if the to-do is performed.  </dd>
<dt>Class <a class="el" href="structflow_1_1log_1_1Msg__metadata.html">flow::log::Msg_metadata</a>  </dt>
<dd><a class="anchor" id="_todo000047"></a>Add support in <a class="el" href="structflow_1_1log_1_1Msg__metadata.html" title="Simple data store containing all of the information generated at every logging call site by flow::log...">Msg_metadata</a> for a message ID which could more straightforwardly help the human log reader to map a log line to the originating log call site in source code. One approach, then, might be to output that message ID when available; else output <a class="el" href="structflow_1_1log_1_1Msg__metadata.html#a5e7224fd18f1b9b2600508ac399687dd" title="Pointer/length into static-storage string that would have come from built-in __FILE__ macro which is ...">m_msg_src_file</a>, <a class="el" href="structflow_1_1log_1_1Msg__metadata.html#a2f17801fae328d1fdc070edc2578dd91" title="Copy of integer that would have come from built-in __LINE__ macro which is auto-invoked by all FLOW_L...">m_msg_src_line</a>, <a class="el" href="structflow_1_1log_1_1Msg__metadata.html#af7704a632434e7b52c43ab6047cf023b" title="Analogous to m_msg_src_file but coming from __FUNCTION__, not __FILE__. See m_msg_src_file perf notes...">m_msg_src_function</a>; or maybe both.  </dd>
<dt>Class <a class="el" href="classflow_1_1log_1_1Serial__file__logger.html">flow::log::Serial_file_logger</a>  </dt>
<dd><a class="anchor" id="_todo000041"></a>Consider having <a class="el" href="classflow_1_1log_1_1Serial__file__logger.html" title="An internal-use implementation of Logger that logs messages to a given file-system path,...">Serial_file_logger</a> internally buffer any attempted log requests that it couldn't write to file due to I/O error. The logic already attempts re-open repeatedly but doesn't attempt to re-log failed lines.  </dd>
<dt>Class <a class="el" href="structflow_1_1net__flow_1_1Ack__packet.html">flow::net_flow::Ack_packet</a>  </dt>
<dd><a class="anchor" id="_todo000066"></a>Conceivably, since <a class="el" href="structflow_1_1net__flow_1_1Ack__packet.html" title="Internal net_flow struct that encapsulates the Flow-protocol low-level ACK packet.">Ack_packet</a> actually stores 1+ acknowledgments, it should become Acks_packet, of type ACKS (not ACK). Many comments and log messages would become clearer, as no one would assume an individual packet's acknowledgment when reading "an ACK" or similar phrase.  </dd>
<dt>Member <a class="el" href="structflow_1_1net__flow_1_1Ack__packet.html#a374da5eef60c7d8a4085b41cf7b2adfe">flow::net_flow::Ack_packet::ack_delay_t</a>  </dt>
<dd><a class="anchor" id="_todo000070"></a>Reconsider the encoding width. If <code>Ack_delay_time_unit(1)</code> is a nanosecond, then 32 bits would support a maximum delay of ~4.1 seconds which is likely fine for most real-world scenarios. This would reduce the size of ACK packets quite a bit.  </dd>
<dt>Class <a class="el" href="classflow_1_1net__flow_1_1asio_1_1Node.html">flow::net_flow::asio::Node</a>  </dt>
<dd><p class="startdd"><a class="anchor" id="_todo000057"></a>To enable reactor-style <code>async_*()</code> operations, meaning waiting for readability/writability/etc. but <em>not</em> performing the actual operation before calling the user handler, we provide a <code>nullptr_t</code>-style interface; like newer boost.asio versions we should deprecate this in favor of simpler <code>async_wait()</code> APIs. This would apply to <a class="el" href="classflow_1_1net__flow_1_1asio_1_1Peer__socket.html" title="A net_flow::Peer_socket that adds integration with boost.asio.">net_flow::asio::Peer_socket</a> and <a class="el" href="classflow_1_1net__flow_1_1asio_1_1Server__socket.html" title="A net_flow::Server_socket that adds integration with boost.asio.">net_flow::asio::Server_socket</a> APIs. Similarly consider doing this for the <code>sync_*()</code> operations in the non-<code>asio</code> superclasses <a class="el" href="classflow_1_1net__flow_1_1Peer__socket.html" title="A peer (non-server) socket operating over the Flow network protocol, with optional stream-of-bytes an...">net_flow::Peer_socket</a> and <a class="el" href="classflow_1_1net__flow_1_1Server__socket.html" title="A server socket able to listen on a single Flow port for incoming connections and return peer sockets...">net_flow::Server_socket</a>. Note that <a class="el" href="classflow_1_1net__flow_1_1Event__set.html#a43dfb881ab3a9488a6a60dd83ad21c85" title="Moves object to State::S_WAITING state, saves the given handler to be executed later (in a different,...">Event_set::async_wait()</a> and <a class="el" href="classflow_1_1net__flow_1_1Event__set.html#a133c518168fe3922ab6729a99077dd3f" title="Blocks indefinitely until one or more of the previously described events hold – or the wait is interr...">Event_set::sync_wait()</a> already exist; they are powerful but a bit complex to use in these simple situations. Hence the following hypothetical wrappers would be welcome replacements for the deprecated <code>nullptr_t</code> and "reactor-style" APIs in these classes: <code>net_flow::Peer_socket::sync_wait(Event_set::Event_type)</code>, <code>net_flow::asio::Peer_socket::async_wait(Event_set::Event_type)</code>, <code>net_flow::Server_socket::sync_wait()</code>, <code>net_flow::Server_socket::async_wait()</code>. (Timeout-arg versions would be desired also, as they exist now.)</p>
<p class="enddd"><a class="anchor" id="_todo000058"></a>The <code>sync_*()</code> early-destruction functionality explained in "Thread safety of destructor" above could be extended to <code>async_*()</code>. This is not a huge deal but would be nice for consistency. Not allowing it currently is really a result of internal implementation concerns (it is easier to not account for this corner case). Alternatively, maybe we should go the other way and not support the somewhat-weird corner case in <code>sync_*()</code>.  </p>
</dd>
<dt>Class <a class="el" href="classflow_1_1net__flow_1_1Congestion__control__strategy.html">flow::net_flow::Congestion_control_strategy</a>  </dt>
<dd><a class="anchor" id="_todo000059"></a>Tuck away all congestion control-related symbols into new <code>namespace cong_ctl</code>?  </dd>
<dt>Class <a class="el" href="classflow_1_1net__flow_1_1Drop__timer.html">flow::net_flow::Drop_timer</a>  </dt>
<dd><a class="anchor" id="_todo000060"></a><a class="el" href="classflow_1_1net__flow_1_1Drop__timer.html" title="Internal net_flow class that maintains the Drop Timer for DATA packet(s) to have been sent out over a...">Drop_timer</a> has data members that are references to non-<code>const</code>. These should be pointers for max consistency of style with other code in the project (and that style is encouraged for certain good reasons). Scour rest of code for other such data members as well.  </dd>
<dt>Member <a class="el" href="namespaceflow_1_1net__flow_1_1error.html#a18e13de77b119ea0f2be411490859482">flow::net_flow::error::Code</a>  </dt>
<dd><a class="anchor" id="_todo000079"></a>Consider adding specific subclasses of <a class="el" href="classflow_1_1error_1_1Runtime__error.html" title="An std::runtime_error (which is an std::exception) that stores an Error_code.">flow::error::Runtime_error</a> for the key codes, like <code>WAIT_INTERRUPTED</code> and <code>USER_TIMEOUT</code>. In doing so, make sure there's an automated syntactic-sugary way of doing this, so that one need only add a code&lt;-&gt;class pair into some internal centralized mapping, automatically making system emit the proper exception class, defaulting to plain Runtime_error if not specially-mapped.  </dd>
<dt>Member <a class="el" href="classflow_1_1net__flow_1_1Event__set.html#a72b26f25a2e76e3c61b4760b59de5b25">flow::net_flow::Event_set::emit_result_sockets</a>  (Sockets *target_set, Event_type ev_type, Error_code *err_code=0)</dt>
<dd><a class="anchor" id="_todo000082"></a><a class="el" href="classflow_1_1net__flow_1_1Event__set.html#a72b26f25a2e76e3c61b4760b59de5b25" title="Gets the sockets that satisfy the condition of the given Event_type detected during the last wait.">Event_set::emit_result_sockets()</a> sets a <a class="el" href="classflow_1_1net__flow_1_1Event__set.html#a592b3ce6602051bd9994216041a3cbdc" title="A set of sockets of one type, used to communicate sets of desired and resulting events in various Eve...">Sockets</a> structure which stores <code>boost:any</code>s each of which stores either a <code><a class="el" href="classflow_1_1util_1_1Shared__ptr__alias__holder.html#aa4f1af88393042e60159a3a67f0accbc" title="Short-hand for ref-counted pointer to mutable values of type Target_type::element_type (a-la T*).">Peer_socket::Ptr</a></code> or a <code><a class="el" href="classflow_1_1util_1_1Shared__ptr__alias__holder.html#aa4f1af88393042e60159a3a67f0accbc" title="Short-hand for ref-counted pointer to mutable values of type Target_type::element_type (a-la T*).">Server_socket::Ptr</a></code>; <a class="el" href="classflow_1_1net__flow_1_1Event__set.html#a592b3ce6602051bd9994216041a3cbdc" title="A set of sockets of one type, used to communicate sets of desired and resulting events in various Eve...">Sockets</a> should be changed to store C++17 <code>std::variant</code>s. Performance, both internally and externally, would improve by using this type-safe compile-time mechanism (which is akin to <code>union</code>s but much more pleasant to use). At the time this feature was written, Flow was in C++11, so <code>variant</code>s were not available, and the author wanted to play around with <code>any</code>s instead of haxoring old-school <code>union</code>s. <code>variant</code> is much nicer, however, and the dynamic nature of <code>any</code> is entirely unnecessary here.  </dd>
<dt>Member <a class="el" href="classflow_1_1net__flow_1_1Event__set.html#a9da325d6cb993b975eec99adaafd3dce">flow::net_flow::Event_set::Event_handler</a>  </dt>
<dd><a class="anchor" id="_todo000081"></a>Perhaps <a class="el" href="classflow_1_1net__flow_1_1Event__set.html#a43dfb881ab3a9488a6a60dd83ad21c85" title="Moves object to State::S_WAITING state, saves the given handler to be executed later (in a different,...">async_wait()</a> and other APIs of <a class="el" href="classflow_1_1net__flow_1_1Event__set.html" title="A user-set collection of sockets and desired conditions on those sockets (such as: &quot;socket has data t...">Event_set</a> taking handlers should be templated on handler type for syntactic sugar. Though, currently, this would be no faster (it is internally stored as this type anyway and must be so), nor would it actually improve call code which needs no explicit cast (an object of this type will implicitly be substituted as a conversion from whatever compatible-with-this-signature construction they used). So on balance, this currently appears superior. After all writing non-template bodies is easier/nicer.  </dd>
<dt>Member <a class="el" href="classflow_1_1net__flow_1_1Event__set.html#a47e4e6d304a6a381e7469fe87c547738">flow::net_flow::Event_set::Mutex</a>  </dt>
<dd><a class="anchor" id="_todo000083"></a>This doc header for <a class="el" href="classflow_1_1net__flow_1_1Event__set.html#a47e4e6d304a6a381e7469fe87c547738" title="Short-hand for reentrant mutex type.">Event_set::Mutex</a> should specify what specific behavior requires mutex reentrance, so that for example one could reevaluate whether there's a sleeker code pattern that would avoid it.  </dd>
<dt>Member <a class="el" href="classflow_1_1net__flow_1_1Event__set.html#a592b3ce6602051bd9994216041a3cbdc">flow::net_flow::Event_set::Sockets</a>  </dt>
<dd><a class="anchor" id="_todo000080"></a>Is it necessary to have <a class="el" href="classflow_1_1net__flow_1_1Event__set.html#a592b3ce6602051bd9994216041a3cbdc" title="A set of sockets of one type, used to communicate sets of desired and resulting events in various Eve...">Event_set::Sockets</a> be aliased to <a class="el" href="classflow_1_1util_1_1Linked__hash__set.html" title="An object of this class is a set that combines the lookup speed of an unordered_set&lt;&gt; and ordering an...">util::Linked_hash_set</a>? <code>unordered_set</code> would also work and take somewhat less memory and computational overhead. It would become unordered, instead of ordered chronologically, but that seems like a price possibly worth paying. </dd>
<dt>Class <a class="el" href="structflow_1_1net__flow_1_1Low__lvl__packet.html">flow::net_flow::Low_lvl_packet</a>  </dt>
<dd><a class="anchor" id="_todo000065"></a>With C++11, some lines of code could be eliminated by using <code>using</code> for <a class="el" href="structflow_1_1net__flow_1_1Low__lvl__packet.html" title="Internal net_flow struct that encapsulates the Flow-protocol low-level packet structure and serves as...">Low_lvl_packet</a> sub-types to inherit the constructor(s) of <a class="el" href="structflow_1_1net__flow_1_1Low__lvl__packet.html" title="Internal net_flow struct that encapsulates the Flow-protocol low-level packet structure and serves as...">Low_lvl_packet</a>. (For some of the sub-types an explicit constructor would still be necessary, but it would just shadow the inherited one, which is fine and still saves lines in the other sub-types.) However, for now I've left it as-is, partially to be friendly to the Doxygen documentation generator, and partially to make the interface easy to read. Still, it may be better the other way.  </dd>
<dt>Member <a class="el" href="structflow_1_1net__flow_1_1Low__lvl__packet.html#ab735867345855ad43e5e3ff788cf5a9a">flow::net_flow::Low_lvl_packet::Const_buffer</a>  </dt>
<dd><a class="anchor" id="_todo000067"></a>Consider moving alias <a class="el" href="structflow_1_1net__flow_1_1Low__lvl__packet.html#ab735867345855ad43e5e3ff788cf5a9a" title="Short-hand for boost.asio immutable buffer, which essentially is a pointer to and length of a memory ...">Low_lvl_packet::Const_buffer</a> to <code>util</code> namespace or even outside it, as it is used commonly (or <code>boost::asio::const_buffer</code> is used where <code>Const_buffer</code> could be used for readability).  </dd>
<dt>Class <a class="el" href="classflow_1_1net__flow_1_1Net__env__simulator.html">flow::net_flow::Net_env_simulator</a>  </dt>
<dd><a class="anchor" id="_todo000087"></a>One thing we should probably add for a bit of realism is the following. If a loss range [A, B] is specified, an individual packet's simulated latency will be a uniformly random number in that range. Because of this there will be frequent re-ordering of packets: if range is [50, 100] ms, and we physically get packet X at time 0 and packet Y at time 5 ms, and the simulated latencies are 80, 60 ms, respectively, then Y will "arrive" 80 - 60 - 5 = 15 ms BEFORE X despite being sent earlier (assuming perfect actual network conditions under the simulation). In reality, packets aren't re-ordered all that often. So if we wanted to simulate a high latency without re-ordering, we'd have to set the range to [A, A] for some A. To avoid having to do that, we can add some internal memory into <a class="el" href="classflow_1_1net__flow_1_1Net__env__simulator.html" title="Objects of this class can be fed to Node to make it internally simulate network conditions like loss,...">Net_env_simulator</a> that would put a floor on a randomly generated latency. In the previous example, the range for packet Y would be chosen as [75, 100] (instead of [50, 100]) to guarantee that packet Y arrives at least a bit later than packet X. Of course that might skew the latency range in bursty traffic, but that might be an OK behavior. We could add a knob for how often to ignore this new logic and allow a re-ordering, so that that is also simulated.  </dd>
<dt>Class <a class="el" href="classflow_1_1net__flow_1_1Node.html">flow::net_flow::Node</a>  </dt>
<dd><p class="startdd"><a class="anchor" id="_todo000088"></a><a class="el" href="namespaceflow_1_1net__flow.html" title="Flow module containing the API and implementation of the Flow network protocol, a TCP-inspired stream...">flow::net_flow</a> should use <a class="el" href="namespaceflow_1_1cfg.html" title="Flow module that facilitates configuring modules, such as applications and APIs, via statically and/o...">flow::cfg</a> for its socket-options mechanism. It is well suited for that purpose, and it uses some ideas from those socket-options in the first place but is generic and much more advanced. Currently <code><a class="el" href="namespaceflow_1_1net__flow.html" title="Flow module containing the API and implementation of the Flow network protocol, a TCP-inspired stream...">net_flow</a></code> socket-options are custom-coded from long before <a class="el" href="namespaceflow_1_1cfg.html" title="Flow module that facilitates configuring modules, such as applications and APIs, via statically and/o...">flow::cfg</a> existed.</p>
<p class="interdd"><a class="anchor" id="_todo000096"></a>Receive UDP datagrams as soon as possible (avoid internal buffer overflow): APPROACH 3: Suppose we take APPROACH 1 (no-brainer) plus APPROACH 2. Certain decisions in the latter were made for certain stated reasons, but let's examine those more closely. First note that the APPROACH 1 part will ensure that, given a burst of incoming datagrams, the first UDP <code>recv()</code> will occur somewhere inside boost.asio, so that's definitely a success. Furthermore, strand S will invoke <code>m_low_lvl_sock.async_send()</code> as soon as the pacing module decides to do so; if I recall correctly, boost.asio will invoke the UDP <code><a class="el" href="classflow_1_1net__flow_1_1Node.html#aa19d1f49638ef8ce8883e7477a06d3fc" title="Implementation of non-blocking sock-&gt;send() for socket sock in all cases except when sock-&gt;state() ==...">send()</a></code> right then and there, synchronously (at least I wrote that unequivocally in a <a class="el" href="classflow_1_1net__flow_1_1Node.html#a9f302fe63e36f72931e1a759b0aaaed4" title="Takes given low-level packet structure, serializes it, and initiates asynchronous send of these data ...">Node::async_low_lvl_packet_send_impl()</a> comment). Again, that's as good as we can possibly want. Finally, for messages 2, 3, ... in that incoming datagram burst, our handler will (indirectly but synchronously) perform the UDP <code>recv()</code>s in strand S2. Here we're somewhat at boost.asio's mercy, but assuming its strand task scheduling is as efficient as possible, it should occur on the thread that's free, and either W1 or W2 should be free given the rest of the design. Still, boost.asio docs even say that different strands' tasks are NOT guaranteed to be invoked concurrently (though common sense implies they will be when possible... but WHAT IF WE'RE WRONG!!!?). Also, we don't know how much computational overhead is involved in making strands work so nicely (again, hopefully it's well written... but WHAT IF!!!?). A negative is the small mutex section around the two <a class="el" href="classflow_1_1net__flow_1_1Node.html#a9b9e0224c17190e99debdfd81640bf89" title="The UDP socket used to receive low-level packets (to assemble into application layer data) and send t...">m_low_lvl_sock</a> calls; not complex and probably hardly a performance concern, but still, it's a small cost. Finally, using strands &ndash; while not ugly &ndash; does involve a bit more code, and one has to be careful not to forget to wrap each handler with the appropriate strand (there is no compile- or runtime error if we forget!) So what can we change about APPROACH 2 to avoid those negatives? As stated in that approach's description, we could have thread W not deal with <a class="el" href="classflow_1_1net__flow_1_1Node.html#a9b9e0224c17190e99debdfd81640bf89" title="The UDP socket used to receive low-level packets (to assemble into application layer data) and send t...">m_low_lvl_sock</a> at all; thread W2 would have a separate <code>Task_engine</code> handling only <a class="el" href="classflow_1_1net__flow_1_1Node.html#a9b9e0224c17190e99debdfd81640bf89" title="The UDP socket used to receive low-level packets (to assemble into application layer data) and send t...">m_low_lvl_sock</a> (so no mutex needed). W2 would do both sends and receives on the socket; and absolutely nothing else (to ensure it's as efficient as possible at getting datagrams off the kernel buffer, solving the original problem). Yes, the receiving would have to share time with the sends, but assuming nothing else interferes, this feels like not much of a cost (at least compared with all the heavy lifting thread W does today anyway). Each receive would read off all available messages into raw buffers and pass those (sans any copying) on to thread W via <code>post(W)</code>. The other negative, also already mentioned, is that once pacing module (in thread W) decides that a datagram should be sent, the <code>post(W2)</code> for the task that would peform the send introduces a delay between the decision and the actual UDP <code><a class="el" href="classflow_1_1net__flow_1_1Node.html#aa19d1f49638ef8ce8883e7477a06d3fc" title="Implementation of non-blocking sock-&gt;send() for socket sock in all cases except when sock-&gt;state() ==...">send()</a></code> done by boost.asio. Thinking about it now, it is questionable to me how much of a cost that really is. Without CPU contention, we can measure it; I expect it to be quite cheap, but I coudl be wrong. With CPU contention &ndash; which would have to come from many datagrams arriving at the same time &ndash; I'm not sure. It wouldn't be overly hard to test; basically flood with UDP traffic over loopback and log the delay between W deciding to send datagram and W2 calling <code>m_low_lvl_sock.async_send_to()</code> (obviously use <a class="el" href="namespaceflow.html#a8f2e48761f9ca3ffcaa29872078bbf00" title="Clock used for delicate time measurements, such that the now() method gets the current time relative ...">Fine_clock</a>!). All in all, if we name the dedicated thread approach described here as APPROACH 3, then APPROACH 3 is appealingly simpler than APPROACH 2; and in most ways appears like it'd be at least as efficient and good at solving the original problem as APPROACH 2. The only danger that worries me is this business with messing up pacing (and no, moving pacing into W2 just endangers the receiving efficiency and introduces thread safety problems and complexity) by having it compete with receiving during incoming-traffic-heavy times. Ultimately, I'd recommend timing this "danger zone" as described a bit earlier (also time delay introduced by <code>post(W2)</code> without any traffic coming in); and if it looks good, do APPROACH 3. Otherwise spring for APPROACH 2.</p>
<p class="interdd"><a class="anchor" id="_todo000089"></a><code>ostream</code> output operators for <a class="el" href="classflow_1_1net__flow_1_1Node.html" title="An object of this class is a single Flow-protocol networking node, in the sense that: (1) it has a di...">Node</a> and <a class="el" href="classflow_1_1net__flow_1_1asio_1_1Node.html" title="A subclass of net_flow::Node that adds the ability to easily and directly use net_flow sockets in gen...">asio::Node</a> should exist. Also scour through all types; possibly some others could use the same. (I have been pretty good at already implementing these as-needed for logging; but I may have "missed a spot.")</p>
<p class="interdd"><a class="anchor" id="_todo000090"></a>Some of the <code>ostream&lt;&lt;</code> operators we have take <code>X*</code> instead of <code>const X&amp;</code>; this should be changed to the latter for various minor reasons and for consistency.</p>
<p class="interdd"><a class="anchor" id="_todo000091"></a>Actively support IPv6 and IPv4, particularly in dual-stack mode (wherein <a class="el" href="classflow_1_1net__flow_1_1Server__socket.html" title="A server socket able to listen on a single Flow port for incoming connections and return peer sockets...">net_flow::Server_socket</a> would bind to an IPv6 endpoint but accept incoming V4 and V6 connections alike). It already supports it nominally, in that one can indeed listen on either type of address and connect to either as well, but how well it works is untested, and from some outside experience it might involve some subtle provisions internally.</p>
<p class="interdd"><a class="anchor" id="_todo000092"></a>Based on some outside experience, there maybe be problems &ndash; particularly considering the to-do regarding dual-stack IPv6/v4 support &ndash; in servers listening in multiple-IP situations; make sure we support these seamlessly. For example consider a machine with a WAN IP address and a LAN (10.x.x.x) IP address (and possibly IPv6 versions of each also) that (as is typical) binds on all of them at ANY:P (where P is some port; and ANY is the IPv6 version, with dual-stack mode ensuring V4 datagrams are also received). If a client connects to a LAN IP, while in our return datagrams we set the source IP to the default, does it work? Outside experience shows it might not, depending, plus even if in our protocol it does, it might be defeated by some firewall... the point is it requires investigation (e.g., mimic TCP itself; or look into what IETF or Google QUIC does; and so on).</p>
<p class="interdd"><a class="anchor" id="_todo000093"></a>Receive UDP datagrams as soon as possible (avoid internal buffer overflow): The OS UDP net-stack buffers arriving datagrams until they're <code>recv()</code>d by the application layer. This buffer is limited; on my Linux test machine the default appears to buffer ~80 1k datagrams. With a high sender CWND and high throughput (e.g., over loopback), thread W &ndash; which both reads off UDP datagrams and handles them, synchronously &ndash; cannot always keep up, and the buffer fills up. This introduces Flow loss despite the fact that the datagram actually safely got to the destination; and this is with just ONE sender; in a server situation there could be thousands. In Linux I was able to raise, via <code>setsockopt()</code>, the buffer size to somewhere between 1000 and 2000 1k datagrams. This helps quite a bit. However, we may still overflow the buffer in busy situations (I have seen it, still with only one connection). So, the to-do is to solve this problem. See below to-dos for ideas. WARNING: UDP buffer overflow can be hard to detect and may just look like loss; plus the user not reading off the Receive buffer quickly enough will also incur similar-looking loss. If there were a way to detect the total # of bytes or datagrams pending on a socket, that would be cool, but <code>sock.available()</code> (where <code>sock</code> is a UDP socket) just gives the size of the first queued datagram. Congestion control (if effective) should prevent this problem, if it is a steady-state situation (i.e., loss or queueing delay resulting from not keeping up with incoming datagrams should decrease CWND). However, if it happens in bursts due to high CPU use in the environment, then that may not help. NOTE 1: In practice a <a class="el" href="classflow_1_1net__flow_1_1Node.html" title="An object of this class is a single Flow-protocol networking node, in the sense that: (1) it has a di...">Node</a> with many connections is running on a server and thus doesn't receive that much data but rather sends a lot. This mitigates the UDP-receive-buffer-overflow problem, as the <a class="el" href="classflow_1_1net__flow_1_1Node.html" title="An object of this class is a single Flow-protocol networking node, in the sense that: (1) it has a di...">Node</a> receiving tons of data is more likely to be a client and thus have only one or two connections. Hence if we can handle a handful of really fast flows without such loss, we're good. (On other hand, ACKs are also traffic, and server may get a substantial amount of them. Much testing is needed.) Packet pacing on the sender side may also avoid this loss problem; on the other hand it may also decrease throughput. NOTE 2: Queue delay-based congestion control algorithms, such as FastTCP and Vegas, are highly sensitive to accurate RTT (round trip time) readings. Heavy CPU load can delay the recording of the "received" time stamp, because we call UDP <code>recv()</code> and handle the results all in one thread. Any solution, such as the dedicated thread proposed below, would <em>allow</em> one to record the time stamp immediately upon receipt of the packet by the dedicated thread; while W would independently handle everything else. Is that a good idea though? Maybe not. If the CPU load is such that ACK-receiver-side can't get to the time-stamp-saving, RTT-measuring step without tricks like doing it immediately upon some low-level datagram receipt hook, then that CPU-pegged jungle is, in a way, part of the network and should probably be fairly counted as part of the RTT. So perhaps we should continue to take the RTT time stamp while actually handling the individual acknowledgments. Instead we should strive to use multi-core resources efficiently, so that the gap between receipt (on whatever thread) and acknowledgment processing (on whatever thread) is as small as possible. Then RTT is RTT, but we make it smaller via improved performance. Meanwhile, we hopefully also solve the original problem (internal kernel buffer overflowing and dropping datagrams).</p>
<p class="interdd"><a class="anchor" id="_todo000094"></a>Receive UDP datagrams as soon as possible (avoid internal buffer overflow): APPROACH 1 (CO-WINNER!): One approach is to note that, as of this writing, we call <code>m_low_lvl_sock.async_wait(wait_read)</code>; our handler is called once there is at least 1 message TO read; and then indeed our handler does read it (and any more messages that may also have arrived). Well, if we use actual <code>async_receive()</code> and an actual buffer instead, then boost.asio will read 1 (and no more, even if there are more) message into that buffer and have it ready in the handler. Assuming the mainstream case involves only 1 message being ready, and/or assuming that reading at least 1 message each time ASAP would help significantly, this may be a good step toward relieving the problem, when it exists. The code becomes a tiny bit less elegant, but that's negligible. This seems like a no-brainer that should be included in any solution, but it by itself may not be sufficient, since more than 1 datagram may be waiting, and datagrams 2, 3, ... would still have to be read off by our code in the handler. So other approaches should still be considered.</p>
<p class="interdd"><a class="anchor" id="_todo000095"></a>Receive UDP datagrams as soon as possible (avoid internal buffer overflow): APPROACH 2: To eliminate the problem to the maximum extent possible, we can dedicate its own thread &ndash; call it W2 &ndash; to reading <a class="el" href="classflow_1_1net__flow_1_1Node.html#a9b9e0224c17190e99debdfd81640bf89" title="The UDP socket used to receive low-level packets (to assemble into application layer data) and send t...">m_low_lvl_sock</a>. We could also write to <a class="el" href="classflow_1_1net__flow_1_1Node.html#a9b9e0224c17190e99debdfd81640bf89" title="The UDP socket used to receive low-level packets (to assemble into application layer data) and send t...">m_low_lvl_sock</a> on W2 (which would also allow us to use a different <a class="el" href="namespaceflow_1_1util.html#ac3e89a8a271b0ddc76ac2a0ce488dea4" title="Short-hand for boost.asio event service, the central class of boost.asio.">util::Task_engine</a> from <a class="el" href="classflow_1_1net__flow_1_1Node.html#af5597a417d03e2bed87cd772ef5cef29" title="The main loop engine, functioning in the single-threaded-but-asynchronous callback-based &quot;reactor&quot; st...">m_task_engine</a> to exclusively deal with W2 and the UDP socket <a class="el" href="classflow_1_1net__flow_1_1Node.html#a9b9e0224c17190e99debdfd81640bf89" title="The UDP socket used to receive low-level packets (to assemble into application layer data) and send t...">m_low_lvl_sock</a> &ndash; simpler in some ways that the strand-based solution described below). There are a couple of problems with this. One, it may delay receive ops due to send ops, which compromises the goals of this project in the first place. Two, send pacing code is in thread W (and moving it to W2 would be complex and unhelpful); and once the decision to REALLY send a given datagram has been made, this send should occur right then and there &ndash; queuing that task on W2 may delay it, compromising the quality of send pacing (whose entire nature is about being precise down to the microsecond or even better). Therefore, we'd like to keep <code>m_low_lvl_sock.async_send()</code> in thread W along with all other work (which allows vast majority of internal state to be accessed without locking, basically just the Send/Receive buffers excluded); except the chain of [<code>m_low_lvl_sock.async_receive()</code> -&gt; post handler of received datagram onto thread W; and immediately <code>m_low_lvl_sock.async_receive()</code> again] would be on W2. AS WRITTEN, this is actually hard or impossible to do with boost.asio because of its design: <a class="el" href="classflow_1_1net__flow_1_1Node.html#a9b9e0224c17190e99debdfd81640bf89" title="The UDP socket used to receive low-level packets (to assemble into application layer data) and send t...">m_low_lvl_sock</a> must belong to exactly one <code>Task_engine</code> (here, <a class="el" href="classflow_1_1net__flow_1_1Node.html#af5597a417d03e2bed87cd772ef5cef29" title="The main loop engine, functioning in the single-threaded-but-asynchronous callback-based &quot;reactor&quot; st...">m_task_engine</a>), whereas to directly schedule a specific task onto a specific thread (as above design requires) would require separate <code>Task_engine</code> objects (1 per thread): boost.asio guarantees a task will run on <em>a</em> thread which is currently executing <code>run()</code> &ndash; if 2 threads are executing <code>run()</code> on the same service, it is unknown which thread a given task will land upon, which makes the above design (AS WRITTEN) impossible. (Side note: I'm not sure it's possible in plain C with BSD sockets either. A naive design, at least, might have W <code>select()</code> on <code>m_low_lvl_sock.native()</code> for writability as well other stuff like timers, while W2 <code>select()</code>s on same for readability; then the two threads perform UDP <code><a class="el" href="classflow_1_1net__flow_1_1Node.html#aa19d1f49638ef8ce8883e7477a06d3fc" title="Implementation of non-blocking sock-&gt;send() for socket sock in all cases except when sock-&gt;state() ==...">send()</a></code> and <code>recv()</code>, respectively, when so instructed by <code>select()</code>s. Is it allowed to use <code>select()</code> on the same socket concurrently like that? StackOverflow.com answers are not clear cut, and to me, at least, it seems somewhat dodgy.) However, an equivalent design IS possible (and supported cleanly by boost.asio): In addition to the 2 threads, set up 2 strands, S and S2. All work except the <a class="el" href="classflow_1_1net__flow_1_1Node.html#a9b9e0224c17190e99debdfd81640bf89" title="The UDP socket used to receive low-level packets (to assemble into application layer data) and send t...">m_low_lvl_sock</a> reads and posting of the handler onto S will be scheduled with a strand S. All work regarding <a class="el" href="classflow_1_1net__flow_1_1Node.html#a9b9e0224c17190e99debdfd81640bf89" title="The UDP socket used to receive low-level packets (to assemble into application layer data) and send t...">m_low_lvl_sock</a> reads and posting of its handler onto S will be scheduled with a strand S2. Recall that executing tasks in 1 strand, with 2+ threads executing <code>run()</code>, guarantees that no 2+ of those tasks will execute simultaneously &ndash; always in series. This is actually &ndash; in terms of efficiency and thread safety &ndash; equivalent to the above W/W2 design. Since the S tasks will always execute serially, no locking is necessary to prevent concurrent execution; thus what we know today as thread W tasks (which need no locking against each other) will be equally thread safe; and same holds for the new S2 tasks (which are considerably simpler and fewer in number). So that's the thread safety aspect; but why is efficiency guaranteed? The answer becomes clear if one pictures the original thread W/W2 design; basically little task blocks serially pepper thread W timeline; and same for W2. By doing it with strands S and S2 (running on top of threads W and W2), the only thing that changes is that the little blocks might at random be swapped between threads. So the series of tasks T1 -&gt; T2 -&gt; T3 -&gt; T4 meant for for S might actually jump between W and W2 randomly; but whenever thread W2 is chosen instead of thread W, that leaves an empty "space" in thread W, which will be used by the S2 task queue if it needs to do work at the same time. So tasks may go on either thread, but the strands ensure that both are used with maximum efficiency while following the expected concurrency constraints (that each strand's tasks are to be executed in series). Note, however, that <a class="el" href="classflow_1_1net__flow_1_1Node.html#a9b9e0224c17190e99debdfd81640bf89" title="The UDP socket used to receive low-level packets (to assemble into application layer data) and send t...">m_low_lvl_sock</a> (the socket object) is not itself safe for concurrent access, so we WILL need a lock to protect the tiny/short calls <code>m_low_lvl_sock.async_receive()</code> and <code>m_low_lvl_sock.async_send()</code>: we specifically allow that a read and write may be scheduled to happen simultaneously, since the two are independent of each other and supposed to occur as soon as humanly possible (once the desire to perform either one is expressed by the code &ndash; either in the pacing module in strand S or the read handler in S2). In terms of nomenclature, if we do this, it'd be more fair to call the threads W1 and W2 (as they become completely equal in this design). (In general, any further extensions of this nature (if we want still more mostly-independent task queues to use available processor cores efficiently), we would add 1 strand and 1 worker thread per each such new queue. So basically there's a thread pool of N threads for N mostly-independent queues, and N strands are used to use that pool efficiently without needing to lock any data that are accessed exclusively by at most 1 queue's tasks only. Resources accessed by 2 or more task queues concurrently would need explicit locking (e.g., <a class="el" href="classflow_1_1net__flow_1_1Node.html#a9b9e0224c17190e99debdfd81640bf89" title="The UDP socket used to receive low-level packets (to assemble into application layer data) and send t...">m_low_lvl_sock</a> in this design).) So then where we start thread W today, we'd start the thread pool of 2 threads W1, W2, with each executing <code>m_task_engine.run()</code>. Before the run()s execute, the initial tasks (each wrapped in strand S or S2, as appropriate) need to be posted onto <a class="el" href="classflow_1_1net__flow_1_1Node.html#af5597a417d03e2bed87cd772ef5cef29" title="The main loop engine, functioning in the single-threaded-but-asynchronous callback-based &quot;reactor&quot; st...">m_task_engine</a>; this can even occur in the user thread U in the constructor, before W1 and W2 are created. The destructor would <code>m_task_engine.stop()</code> (like today), ending each thread's <code>run()</code> and trigger the imminent end of that thread; at which point destructor can <code>W1.join()</code> and <code>W2.join()</code> (today it's just <code>W.join()</code>).</p>
<p class="interdd"><a class="anchor" id="_todo000107"></a>Consider removing Flow ports and even <a class="el" href="classflow_1_1net__flow_1_1Server__socket.html" title="A server socket able to listen on a single Flow port for incoming connections and return peer sockets...">Server_socket</a>: As explained above, we add the concept of a large set of available Flow ports within each <a class="el" href="classflow_1_1net__flow_1_1Node.html" title="An object of this class is a single Flow-protocol networking node, in the sense that: (1) it has a di...">Node</a>, and each <a class="el" href="classflow_1_1net__flow_1_1Node.html" title="An object of this class is a single Flow-protocol networking node, in the sense that: (1) it has a di...">Node</a> itself has a UDP port all to itself. So, for example, I could bind a <a class="el" href="classflow_1_1net__flow_1_1Node.html" title="An object of this class is a single Flow-protocol networking node, in the sense that: (1) it has a di...">Node</a> to UDP port 1010, and within that <a class="el" href="classflow_1_1net__flow_1_1Node.html#a3c2cd6a6c328a2d5381308258d35d95d" title="Sets up a server on the given local Flow port and returns Server_socket which can be used to accept s...">listen()</a> on Flow ports 1010 (unrelated to UDP port 1010!) and 1011. In retrospect, though, is that complexity necessary? We could save quite a few lines of code, particularly in the implementation (class <a class="el" href="classflow_1_1net__flow_1_1Port__space.html" title="Internal net_flow class that maintains the available Flow-protocol port space, somewhat similarly to ...">Port_space</a>, for example) and the protocol (extra bytes for Flow source and target ports, for example). (They're fun and pretty lines, but the absence of lines is arguably even prettier albeit less fun. On the other hand, bugs aren't fun, and more code implies a higher probability of bugs, maintenance errors, etc.) The interface would also be a bit simpler; and not just due to fewer items in <a class="el" href="structflow_1_1net__flow_1_1Remote__endpoint.html" title="Represents the remote endpoint of a Flow-protocol connection; identifies the UDP endpoint of the remo...">Remote_endpoint</a> (which would in fact reduce to <a class="el" href="namespaceflow_1_1util.html#af1b28edb7bad743c1c77f6b982cc3196" title="Short-hand for the UDP endpoint (IP/port) type.">util::Udp_endpoint</a> and cease to exist). Consider <a class="el" href="classflow_1_1net__flow_1_1Server__socket.html" title="A server socket able to listen on a single Flow port for incoming connections and return peer sockets...">Server_socket</a>; currently <a class="el" href="classflow_1_1net__flow_1_1Node.html#a3c2cd6a6c328a2d5381308258d35d95d" title="Sets up a server on the given local Flow port and returns Server_socket which can be used to accept s...">listen()</a> takes a <a class="el" href="namespaceflow_1_1net__flow.html#a079d8cd87a68d1c5f04c6bce71f0061d" title="Logical Flow port type (analogous to a UDP/TCP port in spirit but in no way relevant to UDP/TCP).">flow_port_t</a> argument and returns a <a class="el" href="classflow_1_1net__flow_1_1Server__socket.html" title="A server socket able to listen on a single Flow port for incoming connections and return peer sockets...">Server_socket</a> which is listening; calling <a class="el" href="classflow_1_1net__flow_1_1Node.html#ac3c67514c259404de3f27a8cae8b0567" title="Implementation of non-blocking serv-&gt;accept() for server socket serv in all cases except when serv-&gt;s...">accept()</a> (etc.) on the latter yields <a class="el" href="classflow_1_1net__flow_1_1Peer__socket.html" title="A peer (non-server) socket operating over the Flow network protocol, with optional stream-of-bytes an...">Peer_socket</a>, as the other side connects. Without Flow ports, there is no argument to <a class="el" href="classflow_1_1net__flow_1_1Node.html#a3c2cd6a6c328a2d5381308258d35d95d" title="Sets up a server on the given local Flow port and returns Server_socket which can be used to accept s...">listen()</a>; in fact, <a class="el" href="classflow_1_1net__flow_1_1Server__socket.html" title="A server socket able to listen on a single Flow port for incoming connections and return peer sockets...">Server_socket</a> itself is not strictly necessary and could be folded into <a class="el" href="classflow_1_1net__flow_1_1Node.html" title="An object of this class is a single Flow-protocol networking node, in the sense that: (1) it has a di...">Node</a>, with <a class="el" href="classflow_1_1net__flow_1_1Node.html#a3c2cd6a6c328a2d5381308258d35d95d" title="Sets up a server on the given local Flow port and returns Server_socket which can be used to accept s...">listen()</a> becoming essentially something that turns on the "are we listening?" Boolean state, while stop_listening() would turn it off (instead of something like <code>Server_socket::close()</code>). (Important note: That was not an endorsement of removing <a class="el" href="classflow_1_1net__flow_1_1Server__socket.html" title="A server socket able to listen on a single Flow port for incoming connections and return peer sockets...">Server_socket</a>. Arguably it is still a nice abstraction. Removing it would certainly remove some boiler-plate machinery to do with <a class="el" href="classflow_1_1net__flow_1_1Server__socket.html" title="A server socket able to listen on a single Flow port for incoming connections and return peer sockets...">Server_socket</a>'s life cycle, on the other hand. Perhaps it's best to take a two-step appraoch; remove Flow ports first; then after a long time, assuming it becomes clear that nothing like them is going to come back, remove <a class="el" href="classflow_1_1net__flow_1_1Server__socket.html" title="A server socket able to listen on a single Flow port for incoming connections and return peer sockets...">Server_socket</a> as well.) A key question is, of course what would we lose? At first glance, Flow port allows multiple connections on a single UDP-port-taking Flow server, including multiple connections from one client (e.g., with differing connection parameters such as reliability levels among the different connections, or "channels")... but actually that'd still work without Flow ports, assuming the "one client's" multiple connections can bind to different (presumably ephemeral) UDP ports; since the tuple (source host, source UDP port) is still enough to distinguish from the 2+ "channels" of the same "client" connecting to the one Flow <a class="el" href="classflow_1_1net__flow_1_1Node.html" title="An object of this class is a single Flow-protocol networking node, in the sense that: (1) it has a di...">Node</a> (vs. today's tuple: source host, source UDP port, source Flow port, destination Flow port; see <code>struct</code> <a class="el" href="structflow_1_1net__flow_1_1Node_1_1Socket__id.html" title="The data nugget uniquely identifying a peer-to-peer connection from a remote endpoint to a port in th...">Socket_id</a>). However, without Flow ports, it is not possible for one <a class="el" href="classflow_1_1net__flow_1_1Node.html" title="An object of this class is a single Flow-protocol networking node, in the sense that: (1) it has a di...">Node</a> to connect to another <a class="el" href="classflow_1_1net__flow_1_1Node.html" title="An object of this class is a single Flow-protocol networking node, in the sense that: (1) it has a di...">Node</a> twice, as each <a class="el" href="classflow_1_1net__flow_1_1Node.html" title="An object of this class is a single Flow-protocol networking node, in the sense that: (1) it has a di...">Node</a> by definition is on one port. Is this important? Maybe, maybe not; for NAT purposes it can be important to use only 1 port; but that typically applies only to the server, while the client can send packets from anywhere. However, gaming applications can be very demanding and for the most restrictive NAT types might desire only a single port used on both sides. So whether to remove Flow ports is somewhat questionable, now that they exist; but arguably they didn't need to be added in the first place, until they were truly needed. I'd probably leave them alone, since they do exist.</p>
<p class="interdd"><a class="anchor" id="_todo000097"></a>Receive UDP datagrams as soon as possible (avoid internal buffer overflow): APPROACH 4: It now occurs to me how to solve the main questionable part about APPROACH 3. If we fear that the reads and writes in thread W2 may compete for CPU, especially the reads delaying timing-sensitive paced writes, then we can eliminate the problem by taking W2's own <a class="el" href="namespaceflow_1_1util.html#ac3e89a8a271b0ddc76ac2a0ce488dea4" title="Short-hand for boost.asio event service, the central class of boost.asio.">util::Task_engine</a> (which would be separate from <a class="el" href="classflow_1_1net__flow_1_1Node.html#af5597a417d03e2bed87cd772ef5cef29" title="The main loop engine, functioning in the single-threaded-but-asynchronous callback-based &quot;reactor&quot; st...">m_task_engine</a>) and have two equal threads W2' and W2'' start up and then each call <code>Task_engine::run()</code>. Without having to use any strand, this will essentially (as documented in boost.asio's doc overview) establish a thread pool of 2 threads and then perform the receive and send tasks at random on whichever thread(s) is/are available at a particular time. Since there's no strand(s) to worry about, the underlying overhead in boost.asio is probably small, so there's nothing to fear about efficiency. In fact, in this case we've now created 3 separate threads W, W2', W2'', all doing things independently of each other, which is an excellent feature in terms of using multiple cores. Do note we will now need a mutex and very short critical sections around the calls to <code>m_low_lvl_sock::async_receive()</code> and <code>m_low_lvl_sock::async_send()</code>, but as noted before this seems extremely unlikely to have any real cost due to the shortess of critical sections in both threads. If this is APPROACH 4, then I'd say just time how much absolute delay is introduced by a &lsquo;post(W2&rsquo;)<code>or</code>post(W2'')` of the async send call compared to directly making such a call on thread W, as is done today. I suspect it's small, in which case the action is go for APPROACH 4... finally.</p>
<p class="interdd"><a class="anchor" id="_todo000098"></a>Receive UDP datagrams as soon as possible (avoid internal buffer overflow): APPROACH 5 (CO-WINNER!): Actually, the thing I've been dismissing in approaches 2-4, which was to combine the pacing logic with the actual <code>m_low_lvl_sock.async_send()</code> (like today) but in their own dedicated thread, now seems like the best way to solve the one questionable aspect of APPROACH 4. So, APPROACH 4, but: Move the pacing stuff into the task queue associated with threads W2' and W2''. So then these 2 threads/cores will be available for 2 task queues: one for pacing timers + datagram sending over <a class="el" href="classflow_1_1net__flow_1_1Node.html#a9b9e0224c17190e99debdfd81640bf89" title="The UDP socket used to receive low-level packets (to assemble into application layer data) and send t...">m_low_lvl_sock</a> (with mutex); the other for receiving over <a class="el" href="classflow_1_1net__flow_1_1Node.html#a9b9e0224c17190e99debdfd81640bf89" title="The UDP socket used to receive low-level packets (to assemble into application layer data) and send t...">m_low_lvl_sock</a> (with same mutex). Now, the only "delay" is moved to before pacing occurs: whatever minimal time cost exists of adding a queue from thread W to thread W2' or W2'' occurs just before the pacing logic, after which chances are the datagram will be placed on a pacing queue anyway and sent off somewhat later; intuitively this is better than the delay occurring between pacing logic and the actual UDP send. Note, also, that the timing-sensitive pacing logic now gets its own thread/core and should thus work better vs. today in situations when thread W is doing a lot of work. This is even more logical than APPROACH 4 in that sense; the pacing and sending are concern 1 and get their own thread (essentially; really they get either W2' or W2'' for each given task); the receiving is concern 2 and gets its own thread (same deal); and all the rest is concern 3 and remains in its own thread W (until we can think of ways to split that into concerns; but that is another to-do). Only one mutex with 2 very small critical sections, as in APPROACH 4, is used. The only subtlety regarding concurrent data access is in <a class="el" href="classflow_1_1net__flow_1_1Node.html#a2983d0d8a4d4eb5c3402f63f68ef44d8" title="Performs important book-keeping based on the event &quot;DATA packet was sent to destination....">Node::mark_data_packet_sent()</a>, which is called just before <code>m_low_lvl_sock.async_send()</code>, and which finalizes certain aspects of <a class="el" href="structflow_1_1net__flow_1_1Peer__socket_1_1Sent__packet_1_1Sent__when.html" title="Data store to keep timing related info when a packet is sent out.">Peer_socket::Sent_packet::Sent_when</a>; most notably <a class="el" href="structflow_1_1net__flow_1_1Peer__socket_1_1Sent__packet_1_1Sent__when.html#ad513e58f4eb611c3a91d96df26f53f2c" title="The timestamp when the packet is sent out.">Peer_socket::Sent_packet::Sent_when::m_sent_time</a> (used in RTT calculation upon ACK receipt later). This is stored in <a class="el" href="classflow_1_1net__flow_1_1Peer__socket.html#a5ea56c54a98289211fef4a672432a2ad" title="The collection of all In-flight packets, indexed by sequence number and ordered from most to least re...">Peer_socket::m_snd_flying_pkts_by_sent_when</a>, which today is not protected by mutex due to only being accessed from thread W; and which is extremely frequently accessed. So either we protect the latter with a mutex (out of the question: it is too frequently accessed and would quite possibly reduce performance) or something else. Currently I think <a class="el" href="classflow_1_1net__flow_1_1Node.html#a2983d0d8a4d4eb5c3402f63f68ef44d8" title="Performs important book-keeping based on the event &quot;DATA packet was sent to destination....">Node::mark_data_packet_sent()</a> should just be placed onto <a class="el" href="classflow_1_1net__flow_1_1Node.html#af5597a417d03e2bed87cd772ef5cef29" title="The main loop engine, functioning in the single-threaded-but-asynchronous callback-based &quot;reactor&quot; st...">m_task_engine</a> (thread W) via <code>post()</code> but perhaps take all or most of the items to update Sent_when with as arguments, so that they (especially <code>Sent_when::m_sent_time</code>) could be determined in thread W2' or W2'' but written thread-safely in W. (There is no way some other thread W task would mess with this area of <a class="el" href="classflow_1_1net__flow_1_1Peer__socket.html#a5ea56c54a98289211fef4a672432a2ad" title="The collection of all In-flight packets, indexed by sequence number and ordered from most to least re...">Peer_socket::m_snd_flying_pkts_by_sent_when</a> before the proposed <a class="el" href="classflow_1_1net__flow_1_1Node.html#a2983d0d8a4d4eb5c3402f63f68ef44d8" title="Performs important book-keeping based on the event &quot;DATA packet was sent to destination....">mark_data_packet_sent()</a> was able to run; thread W had just decided to send that packet over wire in the first place; so there's no reason to access it until ACK &ndash; much later &ndash; or some kind of socket-wide catastrophe.) All that put together I dub APPROACH 5. Thus, APPROACH 1 + APPROACH 5 seems like the best idea of all, distilling all the trade-offs into the the fastest yet close to simplest approach.</p>
<p class="interdd"><a class="anchor" id="_todo000099"></a>More uniform diagnostic logging: There is much diagnostic logging in the implementation (FLOW_ERROR*(), etc.), but some of it lacks useful info like <code>sock</code> or <code>serv</code> (i.e., the <code>ostream</code> representations of <a class="el" href="classflow_1_1net__flow_1_1Peer__socket.html" title="A peer (non-server) socket operating over the Flow network protocol, with optional stream-of-bytes an...">Peer_socket</a> and <a class="el" href="classflow_1_1net__flow_1_1Server__socket.html" title="A server socket able to listen on a single Flow port for incoming connections and return peer sockets...">Server_socket</a> objects, which include the UDP/Flow endpoints involved in each socket). The messages that do include these have to do so explicitly. Provide some macros to automatically insert this info, then convert the code to use the macros in most places. Note that existing logging is rather exhaustive, so this is not the biggest of deals but would be nice for ease of coding (and detailed logging).</p>
<p class="interdd"><a class="anchor" id="_todo000100"></a>It may be desirable to use not boost.asio's out-of-the-box UDP receive routines but rather extensions capable of some advanced features, such as <code>recvmsg()</code> &ndash; which can obtain kernel receipt time stamps and destination IP address via the <code>cmsg</code> feature. This would tie into various other to-dos listed around here. There is, as of this writing, also a to-do in the top-level <code>flow</code> namespace doc header about bringing some code into a new <code>io</code> namespace/Flow module; this includes the aforementioned <code>recvmsg()</code> wrapper.</p>
<p class="interdd"><a class="anchor" id="_todo000101"></a>It may be desirable to further use <code>recvmmsg()</code> for UDP input; this allows to read multiple UDP datagrams with one call for performance.</p>
<p class="interdd"><a class="anchor" id="_todo000102"></a>By the same token, wrapping <code>sendmsg()</code> and <code>sendmmsg()</code> may allow for futher perf and feature improvements &ndash; in some ways potentially symmetrically to <code>recvmsg()</code> and <code>recvmmsg()</code> respectively. However, as of this writing, I (ygoldfel) see this more of an opportunistic "look into it" thing and not something of active value; whereas <code>recv[m]msg()</code> bring actual features we actively desire for practical reasons.</p>
<p class="interdd"><a class="anchor" id="_todo000103"></a>Send and Receive buffer max sizes: These are set to some constants right now. That's not optimal. There are two competing factors: throughput and RAM. If buffer is too small, throughput can suffer in practice, if the Receiver can't read the data fast enough (there are web pages that show this). Possibly with today's CPUs it's no longer true, but I don't know. If buffer is too large and with a lot of users, a LOT of RAM can be eaten up (however note that a server will likely be mostly sending, not receiving, therefore it may need smaller Receive buffers). Therefore, as in Linux 2.6.17+, the buffer sizes should be adaptively sized. It may be non-trivial to come up with a good heuristic, but we may be able to copy Linux. The basic idea would probably be to use some total RAM budget and divide it up among the # of sockets (itself a constant estimate, or adaptive based on the number of sockets at a given time?). Also, buffer size should be determined on the Receive side; the Send side should make its buffer to be of equal size. Until we implement some sensible buffer sizing, it might be a good idea (for demo purposes with few users) to keep the buffers quite large. However, flow control (receive window) is now implemented and should cope well with momentary Receive buffer exhaustion. Related facts found on the web: In Linux, since a long time ago, Send buffer size is determined by other side's Receive buffer size (probably sent over in the SYN or SYN-ACK as the receive window). Also, in older Linuxes, Receive buffer defaults to 128k but can be manually set. Supposedly the default can lead to low throughput in high-speed (gigabit+) situations. Thus Linux 2.6.17+ apparently made the Receive buffer size adaptive.</p>
<p class="interdd"><a class="anchor" id="_todo000104"></a>Drop Acknowledgments: DCCP, a somewhat similar UDP-based protocol, uses the concept of Data-Dropped acknowledgments. If a packet gets to the receiver, but the receiver is forced to drop it (for example, no Receive buffer space; not sure if there are other reasons in Flow protocol), then the sender will only find out about this by inferring the drop via Drop Timeout or getting acknowledgments for later data. That may take a while, and since receiver-side drops can be quite large, it would be more constructive for the receive to send an un-ACK of sorts: a Data-Dropped packet informing the sender that specific data were dropped. The sender can then update his congestion state (and retransmit if we enable that). See RFC 4340 and 4341.</p>
<p class="interdd"><a class="anchor" id="_todo000105"></a>Add extra-thread-safe convention for setting options: We can provide a thread-safe (against other user threads doing the same thing) macro to set a given option. <a class="el" href="classflow_1_1net__flow_1_1Node.html#a2779cc2429651696df1bca8de73534f3" title="Dynamically replaces the current options set (options()) with the given options set.">set_options()</a> has a documented, albeit in practice not usually truly problematic, thread safety flaw if one calls <a class="el" href="classflow_1_1net__flow_1_1Node.html#a159b9c2b9a8a969f5e3715341c433aec" title="Copies this Node&#39;s option set and returns that copy.">options()</a>, modifies the result, then calls <a class="el" href="classflow_1_1net__flow_1_1Node.html#a2779cc2429651696df1bca8de73534f3" title="Dynamically replaces the current options set (options()) with the given options set.">set_options()</a>. Since another thread may modify the <a class="el" href="classflow_1_1net__flow_1_1Node.html" title="An object of this class is a single Flow-protocol networking node, in the sense that: (1) it has a di...">Node</a>'s options between the two calls, the latter call may unintentionally revert an option's value. Macro would take an option "name" (identifier for the <a class="el" href="structflow_1_1net__flow_1_1Node__options.html" title="A set of low-level options affecting a single Flow Node, including Peer_socket objects and other obje...">Node_options</a> member), a <a class="el" href="classflow_1_1net__flow_1_1Node.html" title="An object of this class is a single Flow-protocol networking node, in the sense that: (1) it has a di...">Node</a>, and a target value for the option and would work together with a helper method template to obtain the necessary lock, make the assignment to the internal option, and give up the lock. The implementation would probably require <a class="el" href="classflow_1_1net__flow_1_1Node.html" title="An object of this class is a single Flow-protocol networking node, in the sense that: (1) it has a di...">Node</a> to expose its internal stored <a class="el" href="structflow_1_1net__flow_1_1Node__options.html" title="A set of low-level options affecting a single Flow Node, including Peer_socket objects and other obje...">Node_options</a> for the benefit of this macro only. Anyway, this feature is not super-important, as long as the user is aware that modifying options from multiple threads simultaneously may result in incorrect settings being applied.</p>
<p class="interdd"><a class="anchor" id="_todo000106"></a>The preceding to-do regarding <a class="el" href="structflow_1_1net__flow_1_1Node__options.html" title="A set of low-level options affecting a single Flow Node, including Peer_socket objects and other obje...">Node_options</a> applies to <a class="el" href="structflow_1_1net__flow_1_1Peer__socket__options.html" title="A set of low-level options affecting a single Peer_socket.">Peer_socket_options</a> stored in <a class="el" href="classflow_1_1net__flow_1_1Peer__socket.html" title="A peer (non-server) socket operating over the Flow network protocol, with optional stream-of-bytes an...">Peer_socket</a> in in an analogous way.</p>
<p class="interdd"><a class="anchor" id="_todo000109"></a>In <a class="el" href="classflow_1_1net__flow_1_1Node.html#a0055a25892990585561e69ceb02214d5" title="Completion handler for async_low_lvl_packet_send_impl(); called when the packet is either successfull...">Node::low_lvl_packet_sent()</a>, the UDP <code>async_send()</code> handler, there is an inline to-do about specially treating the corner case of the <code>would_block</code> and <code>try_again</code> boost.asio errors being reported (a/k/a POSIX <code>EAGAIN</code>/<code>EWOULDBLOCK</code>). Please see that inline comment for details.</p>
<p class="interdd"><a class="anchor" id="_todo000111"></a>Make use of <a class="el" href="classflow_1_1async_1_1Concurrent__task__loop.html" title="The core flow::async interface, providing an optionally multi-threaded thread pool onto which runnabl...">flow::async::Concurrent_task_loop</a> or <a class="el" href="classflow_1_1async_1_1Single__thread__task__loop.html" title="A Concurrent_task_loop-related adapter-style class that represents a single-thread task loop; essenti...">flow::async::Single_thread_task_loop</a>, instead of manually setting up a thread and <a class="el" href="namespaceflow_1_1util.html#ac3e89a8a271b0ddc76ac2a0ce488dea4" title="Short-hand for boost.asio event service, the central class of boost.asio.">util::Task_engine</a>, for <a class="el" href="classflow_1_1net__flow_1_1Node.html#a84b7e03ae7b13dae0f59db1475e46fd3" title="Worker thread (= thread W). Other members should be initialized before this to avoid race condition.">m_worker</a>. I, Yuri, wrote the constructor, <a class="el" href="classflow_1_1net__flow_1_1Node.html#a5b056ed753b832458f44f537acfe86d7" title="Worker thread W (main event loop) body.">worker_run()</a>, destructor, and related setup/teardown code as my very first boost.asio activity ever. It's solid, but <a class="el" href="namespaceflow_1_1async.html" title="Flow module containing tools enabling multi-threaded event loops operating under the asynchronous-tas...">flow::async</a> just makes it easier and more elegant to read/maintain; plus this would increase Flow-wide consistency. It would almost certainly reduce the number of methods and, even nicely, state (such as <a class="el" href="classflow_1_1net__flow_1_1Node.html#aa440f6ff5442c58323bec7a2304b0d90" title="Promise that thread W sets to truthy Error_code if it fails to initialize or falsy once event loop is...">m_event_loop_ready</a>).</p>
<p class="interdd"><a class="anchor" id="_todo000110"></a>Class <a class="el" href="classflow_1_1net__flow_1_1Node.html" title="An object of this class is a single Flow-protocol networking node, in the sense that: (1) it has a di...">Node</a> <code>private</code> section is very large. It's so large that the implementations of the methods therein are split up among different files such as <code>flow_peer_socket.cpp</code>, <code>flow_event_set.cpp</code>, <code>flow_low_lvl_io.cpp</code>, etc. Consider the following scheme to both better express this separation as well as enforce which of a given method group's method(s) are meant to be called by outside code vs. being helpers thereof: Introduce <code>static</code>-method-only inner classes (and, conceivably, even classes within those classes) to enforce this grouping (<code>public</code> methods and <code>private</code> methods enforcing what is a "public" helper vs. a helper's helper).</p>
<p class="enddd"><a class="anchor" id="_todo000108"></a>Multi-core/multi-threading: The existing implementation already has a nice multi-threaded property, namely that each <a class="el" href="classflow_1_1net__flow_1_1Node.html" title="An object of this class is a single Flow-protocol networking node, in the sense that: (1) it has a di...">Node</a> (object that binds to a single UDP endpoint/port) is entirely independent of any other such object &ndash; they have entirely separate data, and each one does all its work on a separate thread. So to make use of multiple cores/processors, one can set up multiple <a class="el" href="classflow_1_1net__flow_1_1Node.html" title="An object of this class is a single Flow-protocol networking node, in the sense that: (1) it has a di...">Node</a> objects. (Obviously this only makes sense for apps where using multiple ports is acceptable or even desired. E.g., a server could listen on N different UDP ports, where N=# of cores.) However, it would be nice for a single <a class="el" href="classflow_1_1net__flow_1_1Node.html" title="An object of this class is a single Flow-protocol networking node, in the sense that: (1) it has a di...">Node</a> to be as multi-core/processor-friendly as possible. This is partially addressed by the "Dedicated thread to receive
UDP datagrams ASAP" to-do item elsewhere. We could go further. boost.asio lets one easily go from 1 thread to multiple threads by simply starting more worker threads like W (W1, W2, ...) and executing <code>m_task_engine::run()</code> in each one &ndash; note that <a class="el" href="classflow_1_1net__flow_1_1Node.html#af5597a417d03e2bed87cd772ef5cef29" title="The main loop engine, functioning in the single-threaded-but-asynchronous callback-based &quot;reactor&quot; st...">m_task_engine</a> is shared (sans lock). Then subsequent handlers (timer-fired handlers, ack-received handlers, data-received handlers, and many more) would be assigned evenly to available threads currently executing run(). However, then all data these handlers access would need to be protected by a mutex or mutexes, which would be a significant increase in complexity and maintenance headaches compared to existing code, which features mutexes for data accessed both by W and the user thread(s) U &ndash; which excludes most <a class="el" href="classflow_1_1net__flow_1_1Node.html" title="An object of this class is a single Flow-protocol networking node, in the sense that: (1) it has a di...">Node</a>, <a class="el" href="classflow_1_1net__flow_1_1Server__socket.html" title="A server socket able to listen on a single Flow port for incoming connections and return peer sockets...">Server_socket</a>, <a class="el" href="classflow_1_1net__flow_1_1Peer__socket.html" title="A peer (non-server) socket operating over the Flow network protocol, with optional stream-of-bytes an...">Peer_socket</a> state &ndash; essentially the user-visible "state" enums, and the Receive and Send buffers; but hugely complex things like the scoreboards, etc. etc., needed no mutex protection, but with this change they would need it. Actually, if the implementation essentially uses one mutex M, and every handler locks it for the entirety of its execution, then one isn't really making use of multi-cores/etc. anyway. One could make use of boost.asio "strands" to avoid the need for the mutex &ndash; just wrap every handler in a shared strand S, and no locking is needed; boost.asio will never execute two handlers simultaneously in different threads. While this would arguably make the code simpler, but in terms of performance it wouldn't make any difference anyway, as it is functionally equivalent to the lock-M-around-every-operation solution (in fact, internally, it might even amount to exactly that anyway). So that's probably not worth it. We need to have more mutexes or strands, based on some other criterion/criteria. After a datagram is demuxed, vast majority of work is done on a particular socket independently of all others. Therefore we could add a mutex (or use an equivalent) into the socket object and then lock on that mutex. Multiple threads could then concurrently handle multiple sockets. However, for this to be used, one would have to use a single <a class="el" href="classflow_1_1net__flow_1_1Node.html" title="An object of this class is a single Flow-protocol networking node, in the sense that: (1) it has a di...">Node</a> (UDP endpoint) with multiple sockets at the same time. Without any changes at all, one can get the same concurrency by instead setting up multiple <a class="el" href="classflow_1_1net__flow_1_1Node.html" title="An object of this class is a single Flow-protocol networking node, in the sense that: (1) it has a di...">Node</a> objects. Other than a bit of lost syntactic sugar (arguably) &ndash; multiple <a class="el" href="classflow_1_1net__flow_1_1Node.html" title="An object of this class is a single Flow-protocol networking node, in the sense that: (1) it has a di...">Node</a> objects needed, each one having to initialize with its own set of options, for example &ndash; this is particularly cost-free on the client side, as each <a class="el" href="classflow_1_1net__flow_1_1Node.html" title="An object of this class is a single Flow-protocol networking node, in the sense that: (1) it has a di...">Node</a> can just use its own ephemeral UDP port. On the server side the network architecture has to allow for multiple non-ephemeral ports, and the client must know to (perhaps randomly) connect to one of N UDP ports/endpoints on the server, which is more restrictive than on the client. So perhaps there are some reasons to add the per-socket concurrency &ndash; but I would not put a high priority on it. IMPORTANT UPDATE: Long after the preceding text was written, <a class="el" href="namespaceflow_1_1async.html" title="Flow module containing tools enabling multi-threaded event loops operating under the asynchronous-tas...">flow::async</a> Flow module was created containing <a class="el" href="classflow_1_1async_1_1Concurrent__task__loop.html" title="The core flow::async interface, providing an optionally multi-threaded thread pool onto which runnabl...">flow::async::Concurrent_task_loop</a> interface. That looks at the general problem of multi-tasking thread pools and what's the user-friendliest-yet-most-powerful way of doing it. While the preceding discussion in this to-do has been left unchanged, one must first familiarize self with <a class="el" href="namespaceflow_1_1async.html" title="Flow module containing tools enabling multi-threaded event loops operating under the asynchronous-tas...">flow::async</a>; and <em>then</em> read the above, both because some of those older ideas might need reevaluation; and because some of those ideas may have been implemented by <a class="el" href="namespaceflow_1_1async.html" title="Flow module containing tools enabling multi-threaded event loops operating under the asynchronous-tas...">flow::async</a> and are now available easily. </p>
</dd>
<dt>Member <a class="el" href="classflow_1_1net__flow_1_1Node.html#ade49e7bc7202a0adf33fec2d93d1b177">flow::net_flow::Node::close_connection_immediately</a>  (const <a class="el" href="structflow_1_1net__flow_1_1Node_1_1Socket__id.html" title="The data nugget uniquely identifying a peer-to-peer connection from a remote endpoint to a port in th...">Socket_id</a> &amp;socket_id, <a class="el" href="classflow_1_1util_1_1Shared__ptr__alias__holder.html#aa4f1af88393042e60159a3a67f0accbc" title="Short-hand for ref-counted pointer to mutable values of type Target_type::element_type (a-la T*).">Peer_socket::Ptr</a> sock, const Error_code &amp;err_code, bool defer_delta_check)</dt>
<dd><a class="anchor" id="_todo000115"></a>Graceful close not yet implemented w/r/t <a class="el" href="classflow_1_1net__flow_1_1Node.html#ade49e7bc7202a0adf33fec2d93d1b177" title="A thread W method that handles the transition of the given socket from OPEN (any sub-state) to CLOSED...">close_connection_immediately()</a>. </dd>
<dt>Member <a class="el" href="classflow_1_1net__flow_1_1Node.html#a10a92f11a2c9f218cdd9a55f282e79b1">flow::net_flow::Node::send_worker</a>  (<a class="el" href="classflow_1_1util_1_1Shared__ptr__alias__holder.html#aa4f1af88393042e60159a3a67f0accbc" title="Short-hand for ref-counted pointer to mutable values of type Target_type::element_type (a-la T*).">Peer_socket::Ptr</a> sock, bool defer_delta_check)</dt>
<dd><a class="anchor" id="_todo000116"></a>Are there other states where sending DATA packets is OK? If so it would be during graceful termination, if we implement it. See <a class="el" href="classflow_1_1net__flow_1_1Node.html#a10a92f11a2c9f218cdd9a55f282e79b1" title="Thread W implemention of send(): synchronously or asynchronously send the contents of sock-&gt;m_snd_buf...">send_worker()</a> for contedt for this to-do. </dd>
<dt>Member <a class="el" href="classflow_1_1net__flow_1_1Node.html#a99836ffb51eaefa4d2cc21770461cd97">flow::net_flow::Node::socket_id</a>  (<a class="el" href="classflow_1_1util_1_1Shared__ptr__alias__holder.html#aef7998db71c60eeb5d1e3d1a97c14886" title="Short-hand for ref-counted pointer to immutable values of type Target_type::element_type (a-la T cons...">Peer_socket::Const_ptr</a> sock)</dt>
<dd><a class="anchor" id="_todo000117"></a>Could make it a <a class="el" href="structflow_1_1net__flow_1_1Node_1_1Socket__id.html" title="The data nugget uniquely identifying a peer-to-peer connection from a remote endpoint to a port in th...">Socket_id</a> constructor instead.  </dd>
<dt>Member <a class="el" href="classflow_1_1net__flow_1_1Node.html#ab2595f47710ddb06cfb22b6710b7286e">flow::net_flow::Node::validate_options</a>  (const <a class="el" href="structflow_1_1net__flow_1_1Node__options.html" title="A set of low-level options affecting a single Flow Node, including Peer_socket objects and other obje...">Node_options</a> &amp;opts, bool init, Error_code *err_code) const</dt>
<dd><a class="anchor" id="_todo000114"></a>Is it necessary to return <code>opts</code> now that we've switched to C++11 or better? </dd>
<dt>Member <a class="el" href="classflow_1_1net__flow_1_1Node.html#a16c53c4848ea976b075a0b56f33eb118">flow::net_flow::Node::~Node</a>  () override</dt>
<dd><p class="startdd"><a class="anchor" id="_todo000112"></a><a class="el" href="classflow_1_1net__flow_1_1Server__socket.html" title="A server socket able to listen on a single Flow port for incoming connections and return peer sockets...">Server_socket</a> objects closed as if by what?</p>
<p class="enddd"><a class="anchor" id="_todo000113"></a>Provide another method to shut down everything gracefully?  </p>
</dd>
<dt>Member <a class="el" href="namespaceflow_1_1net__flow.html#ad9a8fc9f0d8cdfef76158add1f1d1ddc">flow::net_flow::operator&lt;&lt;</a>  (std::ostream &amp;os, <a class="el" href="classflow_1_1net__flow_1_1Peer__socket.html#a9537860c2c2a7380a45a525e5121880f" title="The state of the socket (and the connection from this end&#39;s point of view) for the internal state mac...">Peer_socket::Int_state</a> state)</dt>
<dd><a class="anchor" id="_todo000124"></a>There are a few guys like this which are marked <code>@internal</code> (Doxygen command) to hide from generated public documentation, and that works, but really they should not be visible in the publicly-exported (not in detail/) header source code; so this should be reorganized for cleanliness. The prototypes like this one can be moved to a detail/ header or maybe directly into .cpp that uses them (for those where it's only one). </dd>
<dt>Member <a class="el" href="namespaceflow_1_1net__flow.html#aad968d3e7e9c3f284bea03e799d5fbc0">flow::net_flow::operator&gt;&gt;</a>  (std::istream &amp;is, <a class="el" href="structflow_1_1net__flow_1_1Peer__socket__options.html#a42ce94dbc5d8be28134261c2eb18b245" title="The possible choices for congestion control strategy for the socket.">Peer_socket_options::Congestion_control_strategy_choice</a> &amp;strategy_choice)</dt>
<dd><a class="anchor" id="_todo000118"></a><a class="el" href="structflow_1_1net__flow_1_1Peer__socket__options.html#a42ce94dbc5d8be28134261c2eb18b245" title="The possible choices for congestion control strategy for the socket.">Peer_socket_options::Congestion_control_strategy_choice</a> stream inserter <code>&lt;&lt;</code> and <code>&gt;&gt;</code> operators should use the <a class="el" href="namespaceflow_1_1util.html#a18a29d6a6e65c3b96c52b5cad5971266" title="Deserializes an enum class value from a standard input stream.">flow::util::istream_to_enum()</a> pattern which is much easier than the overwrought old thing in there now involving two <code>map</code>s. Perhaps add a generic <code>enum_to_strings()</code> to provide the body for <a class="el" href="classflow_1_1net__flow_1_1Congestion__control__selector.html#aca9ae5272088f3ef8fb81f40cc85224c" title="Returns a list of strings, called IDs, each of which textually represents a distinct Congestion_contr...">net_flow::Congestion_control_selector::get_ids()</a>.  </dd>
<dt>Class <a class="el" href="classflow_1_1net__flow_1_1Peer__socket.html">flow::net_flow::Peer_socket</a>  </dt>
<dd><p class="startdd"><a class="anchor" id="_todo000121"></a>Closing connection considerations. May implement closing only via timeouts at first (as opposed to explicit closing). Below text refers to <code>close_final()</code> and <code>close_start()</code>, but those are just ideas and may be replaced with timeout, or nothing. At this time, the only closing supported is abrupt close due to error or abrupt close via <a class="el" href="classflow_1_1net__flow_1_1Peer__socket.html#a9ab710704237f912eb219f9b8d574afc" title="Acts as if fatal error error::Code::S_USER_CLOSED_ABRUPTLY has been discovered on the connection.">close_abruptly()</a>.</p>
<p class="interdd"><a class="anchor" id="_todo000122"></a>Rename <code>State</code> and <code>Open_sub_state</code> to <code>Phase</code> and <code>Open_sub_phase</code> respectively; and <code>Int_state</code> to <code>State</code>. Explain difference between phases (application-layer, user-visible and used close to application layer) and states (transport layer, internal).</p>
<p class="enddd"><a class="anchor" id="_todo000123"></a>Look into a way to defeat the need for boiler-plate trickery &ndash; with low but non-zero perf cost &ndash; involving <code>*_socket</code>-vs-<code><a class="el" href="classflow_1_1net__flow_1_1Node.html" title="An object of this class is a single Flow-protocol networking node, in the sense that: (1) it has a di...">Node</a></code> circular references in method templates, such as the way <a class="el" href="classflow_1_1net__flow_1_1Peer__socket.html#af90b656d7e0b7e746b8beb4e24717cec" title="Sends (adds to the Send buffer) the given bytes of data up to a maximum internal buffer size; and asy...">Peer_socket::send()</a> and <a class="el" href="classflow_1_1net__flow_1_1Peer__socket.html#aeb88e9253cc3f52269cb762c8e57e6d0" title="Receives (consumes from the Receive buffer) bytes of data, up to a given maximum cumulative number of...">Peer_socket::receive()</a> internally make <code><a class="el" href="classflow_1_1Function.html">Function</a>&lt;&gt;</code>s before forwarding to the core in <a class="el" href="classflow_1_1net__flow_1_1Node.html" title="An object of this class is a single Flow-protocol networking node, in the sense that: (1) it has a di...">Node</a>. Can this be done with <code>.inl</code> files? Look into how Boost internally uses <code>.inl</code> files; this could inspire a solution... or not.  </p>
</dd>
<dt>Member <a class="el" href="classflow_1_1net__flow_1_1Peer__socket.html#a9ab710704237f912eb219f9b8d574afc">flow::net_flow::Peer_socket::close_abruptly</a>  (Error_code *err_code=0)</dt>
<dd><p class="startdd"><a class="anchor" id="_todo000126"></a><a class="el" href="classflow_1_1net__flow_1_1Peer__socket.html#a9ab710704237f912eb219f9b8d574afc" title="Acts as if fatal error error::Code::S_USER_CLOSED_ABRUPTLY has been discovered on the connection.">close_abruptly()</a> return <code>bool</code> (<code>false</code> on failure)?</p>
<p class="enddd"><a class="anchor" id="_todo000125"></a>Currently this <a class="el" href="classflow_1_1net__flow_1_1Peer__socket.html#a9ab710704237f912eb219f9b8d574afc" title="Acts as if fatal error error::Code::S_USER_CLOSED_ABRUPTLY has been discovered on the connection.">close_abruptly()</a> is the only way for the user to explicitly close one specified socket. All other ways are due to error (or other side starting graceful shutdown, once we implement that). Once we implement graceful close, via <code>close_start()</code> and <code>close_final()</code>, use of <a class="el" href="classflow_1_1net__flow_1_1Peer__socket.html#a9ab710704237f912eb219f9b8d574afc" title="Acts as if fatal error error::Code::S_USER_CLOSED_ABRUPTLY has been discovered on the connection.">close_abruptly()</a> should be discouraged, or it may even be deprecated (e.g., <code><a class="el" href="classflow_1_1net__flow_1_1Node.html" title="An object of this class is a single Flow-protocol networking node, in the sense that: (1) it has a di...">Node</a></code>s lack a way to initiate an abrupt close for a specific socket). </p>
</dd>
<dt>Member <a class="el" href="classflow_1_1net__flow_1_1Peer__socket.html#aad81cc16fc6f0d353fdc1379e1dd4712">flow::net_flow::Peer_socket::info</a>  () const</dt>
<dd><a class="anchor" id="_todo000128"></a>Provide a similar <a class="el" href="classflow_1_1net__flow_1_1Peer__socket.html#aad81cc16fc6f0d353fdc1379e1dd4712" title="Returns a structure containing the most up-to-date stats about this connection.">info()</a> method that loads an existing structure (for structure reuse). </dd>
<dt>Member <a class="el" href="classflow_1_1net__flow_1_1Peer__socket.html#a9537860c2c2a7380a45a525e5121880f">flow::net_flow::Peer_socket::Int_state</a>  </dt>
<dd><a class="anchor" id="_todo000130"></a><a class="el" href="classflow_1_1net__flow_1_1Peer__socket.html#a9537860c2c2a7380a45a525e5121880f" title="The state of the socket (and the connection from this end&#39;s point of view) for the internal state mac...">Peer_socket::Int_state</a> will also include various states on way to a graceful close, once we implement that.  </dd>
<dt>Member <a class="el" href="classflow_1_1net__flow_1_1Peer__socket.html#a36ab27af6886c7699f9965a5eb296cd6">flow::net_flow::Peer_socket::m_node</a>  </dt>
<dd><a class="anchor" id="_todo000131"></a><code>boost::weak_ptr&lt;<a class="el" href="classflow_1_1net__flow_1_1Node.html" title="An object of this class is a single Flow-protocol networking node, in the sense that: (1) it has a di...">Node</a>&gt;</code> would be ideal for this, but of course then <a class="el" href="classflow_1_1net__flow_1_1Node.html" title="An object of this class is a single Flow-protocol networking node, in the sense that: (1) it has a di...">Node</a> would have to (only?) be available via shared_ptr&lt;&gt;.  </dd>
<dt>Member <a class="el" href="classflow_1_1net__flow_1_1Peer__socket.html#ae82958ed6d71be46b96eceaca6214143">flow::net_flow::Peer_socket::m_rcv_packets_with_gaps</a>  </dt>
<dd><a class="anchor" id="_todo000132"></a>The memory use of this structure could be greatly optimized if, instead of storing each individual received packet's metadata separately, we always merged contiguous sequence number ranges. So for example if packet P1, P2, P3 (contiguous) all arrived in sequence, after missing packet P0, then we'd store P1's first sequence number and the total data size of P1+P2+P3, in a single <code>struct</code> instance. Since a typical pattern might include 1 lost packet followed by 100 received packets, we'd be able to cut down memory use by a factor of about 100 in that case (and thus worry much less about the limit). Of course the code would get more complex and potentially slower (but not necessarily significantly).  </dd>
<dt>Member <a class="el" href="classflow_1_1net__flow_1_1Peer__socket.html#a1f8bebf649551131e6ed6e75c6f98d54">flow::net_flow::Peer_socket::m_snd_next_seq_num</a>  </dt>
<dd><a class="anchor" id="_todo000133"></a>Possibly <a class="el" href="classflow_1_1net__flow_1_1Peer__socket.html#a1f8bebf649551131e6ed6e75c6f98d54" title="The sequence number for the start of the data in the next new DATA packet to be sent out.">m_snd_next_seq_num</a> will apply to other packet types than DATA, probably anything to do with connection termination. </dd>
<dt>Member <a class="el" href="classflow_1_1net__flow_1_1Peer__socket.html#ad1e36619c78b15f46576a18678a4ee67">flow::net_flow::Peer_socket::Mutex</a>  </dt>
<dd><a class="anchor" id="_todo000129"></a>This doc header for <a class="el" href="classflow_1_1net__flow_1_1Peer__socket.html#ad1e36619c78b15f46576a18678a4ee67" title="Short-hand for reentrant mutex type.">Peer_socket::Mutex</a> should specify what specific behavior requires mutex reentrance, so that for example one could reevaluate whether there's a sleeker code pattern that would avoid it.  </dd>
<dt>Member <a class="el" href="classflow_1_1net__flow_1_1Peer__socket.html#ad94e37c8cca01f801f13e9e0e0bc857d">flow::net_flow::Peer_socket::options</a>  () const</dt>
<dd><a class="anchor" id="_todo000127"></a>Provide a similar <a class="el" href="classflow_1_1net__flow_1_1Peer__socket.html#ad94e37c8cca01f801f13e9e0e0bc857d" title="Copies this socket&#39;s option set and returns that copy.">options()</a> method that loads an existing structure (for structure reuse). </dd>
<dt>Member <a class="el" href="structflow_1_1net__flow_1_1Peer__socket_1_1Sent__packet_1_1Sent__when.html#a5265c3824ad25d1de4f888fa811dd4e6">flow::net_flow::Peer_socket::Sent_packet::Sent_when::m_order_num</a>  </dt>
<dd><a class="anchor" id="_todo000134"></a>Can we make <a class="el" href="structflow_1_1net__flow_1_1Peer__socket_1_1Sent__packet_1_1Sent__when.html#a5265c3824ad25d1de4f888fa811dd4e6" title="Order number of the packet.">Sent_when::m_order_num</a> and some of its peers const?  </dd>
<dt>Member <a class="el" href="structflow_1_1net__flow_1_1Peer__socket_1_1Sent__packet_1_1Sent__when.html#abc5bf2e627dad5d2dcfb0324853b37f7">flow::net_flow::Peer_socket::Sent_packet::Sent_when::m_sent_cwnd_bytes</a>  </dt>
<dd><p class="startdd"><a class="anchor" id="_todo000136"></a>Can we make <a class="el" href="structflow_1_1net__flow_1_1Peer__socket_1_1Sent__packet_1_1Sent__when.html#abc5bf2e627dad5d2dcfb0324853b37f7" title="The congestion window size (in bytes) that is used when the packet is sent out.">m_sent_cwnd_bytes</a> and some of its peers <code>const</code>? </p>
<p class="enddd"><a class="anchor" id="_todo000135"></a>Why is <a class="el" href="structflow_1_1net__flow_1_1Peer__socket_1_1Sent__packet_1_1Sent__when.html#abc5bf2e627dad5d2dcfb0324853b37f7" title="The congestion window size (in bytes) that is used when the packet is sent out.">m_sent_cwnd_bytes</a> in <a class="el" href="structflow_1_1net__flow_1_1Peer__socket_1_1Sent__packet_1_1Sent__when.html" title="Data store to keep timing related info when a packet is sent out.">Sent_when</a> <code>struct</code> and not directly in <code><a class="el" href="structflow_1_1net__flow_1_1Peer__socket_1_1Sent__packet.html" title="Metadata (and data, if retransmission is on) for a packet that has been sent one (if retransmission i...">Sent_packet</a></code>? Or maybe it should stay here, but <a class="el" href="structflow_1_1net__flow_1_1Peer__socket_1_1Sent__packet_1_1Sent__when.html" title="Data store to keep timing related info when a packet is sent out.">Sent_when</a> should be renamed <code>Send_attempt</code> (or <code>Send_try</code> for brevity)? Yeah, I think that's it. Then <a class="el" href="structflow_1_1net__flow_1_1Peer__socket_1_1Sent__packet.html#a3acccd2fcd0f4a0c5f7f0c2d6ad31c2e" title="Time stamps, order numbers, and other info at the times when the different attempts (including origin...">Sent_packet::m_sent_when</a> &ndash; a container of <a class="el" href="structflow_1_1net__flow_1_1Peer__socket_1_1Sent__packet_1_1Sent__when.html" title="Data store to keep timing related info when a packet is sent out.">Sent_when</a> objects &ndash; would become <code>m_send_tries</code>, a container of <code>Send_try</code> objects. That makes more sense (sentce?!) than the status quo which involves the singular-means-plural strangeness. </p>
</dd>
<dt>Member <a class="el" href="structflow_1_1net__flow_1_1Peer__socket__info.html#a0ddfcfe377cda3266dfc968797ce4c18">flow::net_flow::Peer_socket_info::m_low_lvl_max_buf_size</a>  </dt>
<dd><a class="anchor" id="_todo000085"></a>This is really a <code><a class="el" href="classflow_1_1net__flow_1_1Node.html" title="An object of this class is a single Flow-protocol networking node, in the sense that: (1) it has a di...">Node</a></code>-wide value; we should probably have a <code>Node_info</code> class for such things.  </dd>
<dt>Member <a class="el" href="structflow_1_1net__flow_1_1Peer__socket__info.html#a784bd75264e26acbccd966e48c07dab9">flow::net_flow::Peer_socket_info::m_snd_cong_ctl_in_flight_bytes</a>  </dt>
<dd><a class="anchor" id="_todo000086"></a>Does <a class="el" href="structflow_1_1net__flow_1_1Peer__socket__info.html#a784bd75264e26acbccd966e48c07dab9" title="In congestion control, the current sent data bytes that have been neither acknowledged nor considered...">m_snd_cong_ctl_in_flight_bytes</a> count data queued in the pacing module or truly In-flight data only?  </dd>
<dt>Member <a class="el" href="structflow_1_1net__flow_1_1Peer__socket__options.html#a1bdc21e84373dd96494a76f11603335e">flow::net_flow::Peer_socket_options::m_st_cong_ctl_max_cong_wnd_blocks</a>  </dt>
<dd><a class="anchor" id="_todo000120"></a>Reconsider this value after Receive window feature is implemented.  </dd>
<dt>Member <a class="el" href="structflow_1_1net__flow_1_1Peer__socket__options.html#aeaaf832a38a5e334789e81e214686ea7">flow::net_flow::Peer_socket_options::m_st_max_block_size</a>  </dt>
<dd><a class="anchor" id="_todo000119"></a>max-block-size should be dynamically determined (but overridable by this option). That is a complex topic, however, with considerations such as MTU discovery, ICMP errors, and who knows what else.  </dd>
<dt>Member <a class="el" href="classflow_1_1net__flow_1_1Peer__socket__send__stats.html#a13bf4f61554410f5dcc8928d64212e04">flow::net_flow::Peer_socket_send_stats::m_init_time</a>  </dt>
<dd><a class="anchor" id="_todo000084"></a><a class="el" href="structflow_1_1net__flow_1_1Peer__socket__receive__stats.html" title="A data store that keeps stats about the incoming direction of a Peer_socket connection to another Flo...">Peer_socket_receive_stats</a> also independently stores a similar value, so to save memory put <a class="el" href="classflow_1_1net__flow_1_1Peer__socket__send__stats.html#a13bf4f61554410f5dcc8928d64212e04" title="The time this object was created; should be about equal to when the socket was created.">m_init_time</a> elsewhere.  </dd>
<dt>Member <a class="el" href="classflow_1_1net__flow_1_1Peer__socket__send__stats__accumulator.html#a8dca8497873f98d8318b3fdd2c034c14">flow::net_flow::Peer_socket_send_stats_accumulator::low_lvl_packet_xfer_requested</a>  (const std::type_info &amp;type)</dt>
<dd><a class="anchor" id="_todo000076"></a>It may make sense to serialize a packet as early as possible, so that the size is known at the stage of <a class="el" href="classflow_1_1net__flow_1_1Peer__socket__send__stats__accumulator.html#a8dca8497873f98d8318b3fdd2c034c14" title="Indicates a packet was given to the sending module to send, possibly subject to pacing.">Peer_socket_send_stats_accumulator::low_lvl_packet_xfer_requested()</a> call. Possibly knowing the serialized size this early can help the pacing module make smarter decisions. </dd>
<dt>Class <a class="el" href="classflow_1_1net__flow_1_1Port__space.html">flow::net_flow::Port_space</a>  </dt>
<dd><a class="anchor" id="_todo000071"></a>While the ephemeral port reuse avoidance works well within a single <a class="el" href="classflow_1_1net__flow_1_1Port__space.html" title="Internal net_flow class that maintains the available Flow-protocol port space, somewhat similarly to ...">Port_space</a> lifetime, it does not carry over across different Port_spaces. I.e., if <a class="el" href="classflow_1_1net__flow_1_1Port__space.html" title="Internal net_flow class that maintains the available Flow-protocol port space, somewhat similarly to ...">Port_space</a> is destructed and then constructed, the container of recently used ports starts out empty, so a port may (with some low but sizable probability) be reused. This could be avoided by persisting (e.g., via boost.serialization) those structures to disk (perhaps at a user-supplied <code>boost::filesystem::path</code>) in <code>~Port_space()</code>. This wouldn't work if <a class="el" href="classflow_1_1net__flow_1_1Port__space.html" title="Internal net_flow class that maintains the available Flow-protocol port space, somewhat similarly to ...">Port_space</a> crashed, however. To solve that, we could easily start a thread in <a class="el" href="classflow_1_1net__flow_1_1Port__space.html#ad02a2c57aa6854659621af98b64d3638" title="Constructs the Port_space with all ports available.">Port_space()</a> (and join it in <code>~Port_space()</code>) that would save that state every N seconds. If that's not enough, we can instead have that thread run a boost.asio <code>io_context::run()</code> event loop and <code>post(io_context&amp;)</code> the save operation onto it (from arbitrary threads) each time a new ephemeral port is reserved (with some kind of protection against incessant disk writing built into this worker thread; e.g., don't save if saved in the last N msec).  </dd>
<dt>Class <a class="el" href="structflow_1_1net__flow_1_1Remote__endpoint.html">flow::net_flow::Remote_endpoint</a>  </dt>
<dd><p class="startdd"><a class="anchor" id="_todo000078"></a>There is a sub-case of the immediately preceding to-do that may be performed first without much controversy. That is the case where there is some member method of some type, that is then called by an extremely thin wrapper free function (not even a <code>friend</code>). In fact, <code><a class="el" href="structflow_1_1net__flow_1_1Remote__endpoint.html#ab38246018ee64662d208371c55ffb1bd" title="Free function that returns remote_endpoint.hash(); has to be a free function named hash_value for boo...">hash_value()</a></code> and <a class="el" href="structflow_1_1net__flow_1_1Remote__endpoint.html#a575cefcce9e8b34d6ea1420d6f6756d4" title="Hash value of this Remote_endpoint for unordered_*&lt;&gt;.">Remote_endpoint::hash()</a> are such a pair. Assuming one does not forget Doxygen's <code>relatesalso</code> command, it would be easy and concise to just get rid of the member and simply move its implementation directly into the free function. After all, people are meant to use the free function anyway, so why the middle-man method? In this example, <code><a class="el" href="structflow_1_1net__flow_1_1Remote__endpoint.html#ab38246018ee64662d208371c55ffb1bd" title="Free function that returns remote_endpoint.hash(); has to be a free function named hash_value for boo...">hash_value()</a></code> would directly compute the hash, and <a class="el" href="structflow_1_1net__flow_1_1Remote__endpoint.html#a575cefcce9e8b34d6ea1420d6f6756d4" title="Hash value of this Remote_endpoint for unordered_*&lt;&gt;.">Remote_endpoint::hash()</a> would not exist; but the former would include a Doxygen <code>relatesalso <a class="el" href="structflow_1_1net__flow_1_1Remote__endpoint.html" title="Represents the remote endpoint of a Flow-protocol connection; identifies the UDP endpoint of the remo...">Remote_endpoint</a></code> command to ensure properly linked generated documentation. </p>
<p class="enddd"><a class="anchor" id="_todo000077"></a>The respected Scott Meyers ("Effective C++") would recommend that <a class="el" href="structflow_1_1net__flow_1_1Remote__endpoint.html#a575cefcce9e8b34d6ea1420d6f6756d4" title="Hash value of this Remote_endpoint for unordered_*&lt;&gt;.">Remote_endpoint::hash()</a> and <a class="el" href="structflow_1_1net__flow_1_1Remote__endpoint.html#a8504aeca20f41a8f2b98ae04803a600e" title="Whether lhs is equal to rhs.">Remote_endpoint::operator==()</a> be written as free functions instead of as <code>struct</code> members (see <a href="http://www.drdobbs.com/cpp/how-non-member-functions-improve-encapsu/184401197">http://www.drdobbs.com/cpp/how-non-member-functions-improve-encapsu/184401197</a>). The same would apply to a potentially great number of other non-<code>virtual</code> methods (including operators) of other <code>struct</code>s and classes that could be implemented without <code>friend</code>, so really pursuing this approach could touch quite a few things. I won't repeat Meyers' reasoning; see the link. I find the reasoning mostly compelling. To summarize his final suggestions: An operation on a type should be a free function if ALL of the following holds: it is not <code>virtual</code> by nature; AND: it is <code>&lt;&lt;</code> or <code>&gt;&gt;</code>, and/or its would-be left operand would require type conversions, and/or it can be <em>implemented</em> entirely via the type's publicly exposed interface. We already follow the advice for <code>&lt;&lt;</code> and <code>&gt;&gt;</code> (if only because we apply it to stream ops, and for that it just makes sense, <code>&lt;&lt;</code> especially, since the stream is the left-most operand there, so it <em>has</em> to be a free function in that case &ndash; and thus also for <code>&gt;&gt;</code> for consistency). The type conversion is not a common thing for this library; so that leaves non-<code>virtual</code> operations (which is most of them) that can be implemented via public APIs only (which is probaby common for <code>struct</code>s like <a class="el" href="structflow_1_1net__flow_1_1Remote__endpoint.html" title="Represents the remote endpoint of a Flow-protocol connection; identifies the UDP endpoint of the remo...">Remote_endpoint</a>, though we generally try not to have lots of <code>struct</code> methods in the first place... usually). Before rushing headlong into this project, consider a few more things, though. 1, Doxygen wouldn't pick up the relation between a non-friend free function dealing with <code>struct</code> or class <code>C</code> and <code>C</code> itself; so verbosity/error-proneness would increase by having to add a Doxygen <code>relatesalso</code> special command to each free function; otherwise documentation becomes less readable (and there's no way to enforce this by having Doxygen fail without this being done, somehow). 2, in the <code>C</code>-returning-static-member-of-<code>C</code> pattern, usually in our code it would call a private <code>C</code> constructor, meaning it would require <code>friend</code> to make it a free function, meaning it breaks Meyers' own rule and thus should be left a member. 3, Meyers tends to place very little value on brevity as its own virtue. If you find that following the above rule in some case seems to be significantly increasing code complexity/size, maybe it's best to leave it alone. (I am thinking of <a class="el" href="structflow_1_1net__flow_1_1Low__lvl__packet.html" title="Internal net_flow struct that encapsulates the Flow-protocol low-level packet structure and serves as...">Low_lvl_packet</a> and its sub-types like <a class="el" href="structflow_1_1net__flow_1_1Ack__packet.html" title="Internal net_flow struct that encapsulates the Flow-protocol low-level ACK packet.">Ack_packet</a>: they are <code>struct</code>s but not typical ones, with complex APIs; making those APIs free function when non-<code>virtual</code> sounds like a rather hairy change that "feels" like it would reduce simplicity and may increase size, at least due to all the necessary 'splaining in the comments.) All that said, this is perhaps worth pursuing (in the pursuit of stylistic perfection) &ndash; but surgically and first-do-no-harm-edly. Oh, also, constructors don't count for this and should continue to remain constructors and not some free functions stuff (should be obvious why, but just in case you get any ideas...). </p>
</dd>
<dt>Class <a class="el" href="classflow_1_1net__flow_1_1Send__bandwidth__estimator.html">flow::net_flow::Send_bandwidth_estimator</a>  </dt>
<dd><a class="anchor" id="_todo000075"></a>Look into Google BBR, a recent (as of 2016) state-of-the-art congestion control algorithm that aims to estimate bandwidth among other things. Depending on what what one decides, another bandwidth estimator class could be written. </dd>
<dt>Class <a class="el" href="structflow_1_1net__flow_1_1Send__pacing__data.html">flow::net_flow::Send_pacing_data</a>  </dt>
<dd><p class="startdd"><a class="anchor" id="_todo000061"></a>One way to provide finer pacing, in the situation where we must send more than 1 packet within the minimal schedulable timer period, is to rely on events other than timer clock ticks that occur more frequently. Namely, ACKs will possibly arrive faster that the minimal timer schedulable ticks do. Therefore, when a time slice begins, we can send no packets (instead of all the packets that must be sent in that time slice, as is the case currently). Then over the course of the time slice we can opportunistically &ndash; upon each ACK send the proper number of queued packets up to that point into the time slice (this can be computed accurately, because we can MEASURE time very accurately &ndash; just can't schedule things as accurately). Finally, once the time slice does expire, send off any packets still remaining for that slice. This way we can opportunistically provide finer pacing, where the ACK clock allows it.</p>
<p class="interdd"><a class="anchor" id="_todo000064"></a>My quick look into Google BBR, the recent advanced congestion control algorithm, suggests that the pacing algorithm can be used as part of congestion control rather than something merely applied after congestion control has made the CWND decision. I believe the idea would be this: If we can somehow very reliably determine the available bandwidth, which Google BBR does purport to do, then instead of controlling send rate via CWND, instead one could apply it to this pacing module. It is a slight modification of the algorithm described above: instead of the rate of sending being equal to CWND/SRTT, make it equal to the bandwidth determined through Google BBR. If I understand correctly, Google BBR does maintain a CWND as well, but this is a safety upper bound, in case the computed bandwidth is too high to be practical (in this case CWND will slow down data being delivered to the pacing module &ndash; but, normally, this would not be the case, and the pacing module would be, in practice, the thing controlling the sending rate). Note this is a pretty big conceptual change; in particular, it would make the queue Q grow much larger than what we see currently.</p>
<p class="interdd"><a class="anchor" id="_todo000063"></a>As noted above we perform packet pacing but currently choose to assign a value of 0 bytes to an ACK. That is, while we do preserve the order of DATA and ACK packets &ndash; if both happen to be in the outgoing stream &ndash; we do not delay the sending of the ACK once it is the next packet to be sent out. However, even so, an ACK's sending may be delayed by the pacing applied to DATA packets intermixed with it. Therefore the ACK delay measurement we take here may be incorrect (too low) in that case. This can cause overestimated RTTs on the sender's side. The to-do is to correct the ACK delay value in a given ACK by adding the pacing delay (if any) of the ACK to the individual ACK delays within it. Conceptually this is similar to the <code>m_sent_when</code> value being set when choosing to send a DATA packet and then corrected in the pacing module later. This to-do is not important until we in practice start mixing sending and receiving at the application layer... but still &ndash; it's worth knowing that there is a design bug here.</p>
<p class="enddd"><a class="anchor" id="_todo000062"></a>Obtain the first RTT measurement based on SYN-SYN_ACK or SYN_ACK-SYN_ACK_ACK interval; then SRTT will be available from the start, and packet pacing can be enabled with the very first DATA packet, avoiding the initial (albeit small) burst of DATA packets. This is pretty easy. </p>
</dd>
<dt>Member <a class="el" href="classflow_1_1net__flow_1_1Sequence__number.html#ab83e641aa9497115f6dbaf76b92f73c6">flow::net_flow::Sequence_number::seq_num_delta_t</a>  </dt>
<dd><a class="anchor" id="_todo000072"></a>Allowing overflow by using this definition of <a class="el" href="classflow_1_1net__flow_1_1Sequence__number.html#ab83e641aa9497115f6dbaf76b92f73c6" title="Integer type used to express differences (distances) between Sequence_numbers.">seq_num_delta_t</a> is somewhat unlike the rest of the implementation of <a class="el" href="classflow_1_1net__flow_1_1Sequence__number.html" title="An internal net_flow sequence number identifying a piece of data.">Sequence_number</a> which avoids overflow like the plague. Should we do something like use <code>double</code> or a big-integer type from boost.multiprecision?  </dd>
<dt>Class <a class="el" href="classflow_1_1net__flow_1_1Server__socket.html">flow::net_flow::Server_socket</a>  </dt>
<dd><p class="startdd"><a class="anchor" id="_todo000138"></a>Limit <a class="el" href="classflow_1_1net__flow_1_1Server__socket.html" title="A server socket able to listen on a single Flow port for incoming connections and return peer sockets...">Server_socket</a> listen queue/set length.</p>
<p class="interdd"><a class="anchor" id="_todo000137"></a>Implement <code>Server_socket::close()</code> functionality &ndash; at least the equivalent of <a class="el" href="classflow_1_1net__flow_1_1Peer__socket.html#a9ab710704237f912eb219f9b8d574afc" title="Acts as if fatal error error::Code::S_USER_CLOSED_ABRUPTLY has been discovered on the connection.">Peer_socket::close_abruptly()</a>.</p>
<p class="enddd"><a class="anchor" id="_todo000139"></a>Rename <code>State</code> to <code>Phase</code>, as with similar to-do in class <a class="el" href="classflow_1_1net__flow_1_1Peer__socket.html" title="A peer (non-server) socket operating over the Flow network protocol, with optional stream-of-bytes an...">Peer_socket</a> doc header.  </p>
</dd>
<dt>Member <a class="el" href="classflow_1_1net__flow_1_1Server__socket.html#aeea8554e23c8d743ca4956c772adbf24">flow::net_flow::Server_socket::accept</a>  (Error_code *err_code=0)</dt>
<dd><a class="anchor" id="_todo000140"></a>Reconsider allowing successful <a class="el" href="classflow_1_1net__flow_1_1Server__socket.html#aeea8554e23c8d743ca4956c772adbf24" title="Non-blocking accept: obtain socket for the least recently established not-yet-obtained peer connectio...">accept()</a> in <a class="el" href="classflow_1_1net__flow_1_1Server__socket.html#a8bc789b851677a47d7c0a1aa0c69a997aa27ca0689984ee6c6784a9265a678d42" title="No accept()s are or will be possible, but Node is still finishing up the closing operation.">State::S_CLOSING</a> state? </dd>
<dt>Member <a class="el" href="classflow_1_1net__flow_1_1Server__socket.html#ac36eb28d50496b3f8f1b9790194e825e">flow::net_flow::Server_socket::m_local_port</a>  </dt>
<dd><a class="anchor" id="_todo000142"></a>Make <a class="el" href="classflow_1_1net__flow_1_1Server__socket.html#ac36eb28d50496b3f8f1b9790194e825e" title="See local_port().">m_local_port</a> <code>const</code>?  </dd>
<dt>Member <a class="el" href="classflow_1_1net__flow_1_1Server__socket.html#a15e9fa900dfa1a0a2d693bf8ffb5b07d">flow::net_flow::Server_socket::Mutex</a>  </dt>
<dd><a class="anchor" id="_todo000141"></a>This doc header for <a class="el" href="classflow_1_1net__flow_1_1Server__socket.html#a15e9fa900dfa1a0a2d693bf8ffb5b07d" title="Short-hand for reentrant mutex type.">Server_socket::Mutex</a> should specify what specific behavior requires mutex reentrance, so that for example one could reevaluate whether there's a sleeker code pattern that would avoid it.  </dd>
<dt>Member <a class="el" href="classflow_1_1net__flow_1_1Socket__buffer.html#a89291fb233a05b7d449d9c3ebb802341">flow::net_flow::Socket_buffer::copy_bytes_from_buf_seq</a>  (Const_it *cur_buf_it, size_t *pos_in_buf, size_t to_copy, <a class="el" href="namespaceflow_1_1util.html#af09888b3a8feaff209b6625e493bc338" title="A concrete Blob_with_log_context that compile-time-disables Basic_blob::share() and the sharing API d...">util::Blob</a> *dest_buf, util::Blob::Iterator dest)</dt>
<dd><a class="anchor" id="_todo000074"></a>It would not be crazy to move <a class="el" href="classflow_1_1net__flow_1_1Socket__buffer.html#a89291fb233a05b7d449d9c3ebb802341" title="Helper that copies, to a given raw memory buffer, a given number of bytes from a given Const_buffer_s...">copy_bytes_from_buf_seq()</a> and <a class="el" href="classflow_1_1net__flow_1_1Socket__buffer.html#a45b5068ab3386d99ae7b27047a23ffda" title="Helper that copies, to a given raw memory buffer, a given number of bytes to a given Mutable_buffer_s...">copy_bytes_to_buf_seq()</a> into <a class="el" href="namespaceflow_1_1util.html#af09888b3a8feaff209b6625e493bc338" title="A concrete Blob_with_log_context that compile-time-disables Basic_blob::share() and the sharing API d...">util::Blob</a> to make it more widely available, though as of this writing there is no demand for this: perhaps a buffer-sequence version of <code>util::Blob::emplace_copy()</code> and in the other direction. In that case <code>util::Blob::assign_copy()</code> and all single-buffer <a class="el" href="namespaceflow_1_1util.html#af09888b3a8feaff209b6625e493bc338" title="A concrete Blob_with_log_context that compile-time-disables Basic_blob::share() and the sharing API d...">util::Blob</a> methods should probably be similarly generalized (overloaded). </dd>
<dt>Member <a class="el" href="classflow_1_1net__flow_1_1Socket__buffer.html#a378b7a330d596f81cdf21ea0eb291c11">flow::net_flow::Socket_buffer::Queue</a>  </dt>
<dd><a class="anchor" id="_todo000073"></a>Investigate pros and cons of deque vs. list empirically.  </dd>
<dt>Member <a class="el" href="structflow_1_1net__flow_1_1Syn__ack__packet.html#a1ed3bdee2d19c2adaaf00fda9b5a7ca8">flow::net_flow::Syn_ack_packet::m_rcv_wnd</a>  </dt>
<dd><a class="anchor" id="_todo000069"></a>We could be similar to TCP by opportunistically sending rcv_wnd in other packet types, namely <a class="el" href="structflow_1_1net__flow_1_1Data__packet.html" title="Internal net_flow struct that encapsulates the Flow-protocol low-level DATA packet.">Data_packet</a>. However this would only help in connections with heavy 2-way traffic. Personally I would then prefer to create a new packet type instead, Rcv_wnd_packet, and also implement some generalized "packet combo" scheme which would allow to piggy-back arbitrary packet types together into a single packet; and then we'd dissociate ACK/SYN_ACK/SYN_ACK_ACK from rcv_wnd.  </dd>
<dt>Member <a class="el" href="structflow_1_1net__flow_1_1Syn__packet.html#abde26e74af2154fda517d6125353f305">flow::net_flow::Syn_packet::m_serialized_metadata</a>  </dt>
<dd><a class="anchor" id="_todo000068"></a>Possibly eliminate this, since NetFlow is reliable; the metadata can just be sent explicitly by the user once the connection is established. However, for now reliability is optional.  </dd>
<dt>Class <a class="el" href="classflow_1_1perf_1_1Checkpointing__timer.html">flow::perf::Checkpointing_timer</a>  </dt>
<dd><a class="anchor" id="_todo000143"></a><code>const std::string&amp;</code> to <code>String_view</code> in all of <a class="el" href="namespaceflow_1_1perf.html" title="Flow module containing tools for profiling and optimization.">flow::perf</a> where applicable. Look into <code>string&amp;&amp;</code> also.  </dd>
<dt>Member <a class="el" href="namespaceflow_1_1perf.html#ad61fb186191e4bb824d14601d11a2eb0">flow::perf::Clock_type</a>  </dt>
<dd><a class="anchor" id="_todo000144"></a>Consider adding a system-calendar-clock (a/k/a POSIX time) type to <a class="el" href="namespaceflow_1_1perf.html#ad61fb186191e4bb824d14601d11a2eb0" title="Clock types supported by flow::perf module facilities, perf::Checkpointing_timer in particular.">perf::Clock_type</a>. It would be a cousin of <a class="el" href="namespaceflow_1_1perf.html#ad61fb186191e4bb824d14601d11a2eb0a8cbf506eda585c026dfb5b72c402d342" title="Measures real time (not processor time), using the highest-resolution system clock available that gua...">Clock_type::S_REAL_HI_RES</a>. It would certainly be inferior in terms of resolution/monotonicity/etc., and one would think <code>S_REAL_HI_RES</code> would always be preferable. Nevertheless it would be interesting to "officially" see its characteristics including in particular (1) resolution and (2) its own perf cost especially vs. <code>S_REAL_HI_RES</code> which we know is quite fast itself. This may also help a certain to-do listed as of this writing in the doc header of <a class="el" href="namespaceflow_1_1log.html" title="Flow module providing logging functionality.">flow::log</a> <a class="el" href="log_8hpp.html#a5daa2b6d16edea74bb8bddc75f7fb801" title="Identical to FLOW_LOG_WITH_CHECKING() but foregoes the filter (Logger::should_log()) check.">FLOW_LOG_WITHOUT_CHECKING()</a> (the main worker bee of the log system, the one that generates each log time stamp).  </dd>
<dt>Member <a class="el" href="namespaceflow_1_1perf.html#a129032bde1082e8f2681669acf6bf03d">flow::perf::operator*=</a>  (<a class="el" href="structflow_1_1perf_1_1Duration__set.html" title="Convenience wrapper around an array&lt;Duration, N&gt;, which stores a duration for each of the N possible ...">Duration_set</a> &amp;target, uint64_t mult_scale)</dt>
<dd><a class="anchor" id="_todo000145"></a>Maybe allow <code>operator*=(Duration_set)</code> by a potentially negative number; same for division. </dd>
<dt>Member <a class="el" href="namespaceflow_1_1perf.html#a0dee22ec7bf250bbbf556a7e15c1f784">flow::perf::timed_function</a>  (Clock_type clock_type, Accumulator *accumulator, Func &amp;&amp;function)</dt>
<dd><p class="startdd"><a class="anchor" id="_todo000147"></a><a class="el" href="namespaceflow_1_1perf.html#a0dee22ec7bf250bbbf556a7e15c1f784" title="Constructs a closure that times and executes void-returning function(), adding the elapsed time with ...">timed_function()</a> overload exists for a single <code>Clock_type</code>, but simultaneous multi-clock timing using the <a class="el" href="namespaceflow_1_1perf.html#aabd93a8a40158585ed149742d8937601" title="Short-hand for a bit-set of N bits which represents the presence or absence of each of the N possible...">perf::Clock_types_subset</a> paradigm (as used, e.g., in <a class="el" href="classflow_1_1perf_1_1Checkpointing__timer.html" title="The central class in the perf Flow module, this efficiently times the user&#39;s operation,...">Checkpointing_timer</a>) would be a useful and consistent API. E.g., one could measure user and system elapsed time simultaneously. As of this writing this only does not exist due to time constraints: a perf-niggardly version targeting one clock type was necessary.</p>
<p class="enddd"><a class="anchor" id="_todo000146"></a><a class="el" href="namespaceflow_1_1perf.html#a0dee22ec7bf250bbbf556a7e15c1f784" title="Constructs a closure that times and executes void-returning function(), adding the elapsed time with ...">timed_function()</a>, when operating on an <code>atomic&lt;duration_rep_t&gt;</code>, uses <code>+=</code> for accumulation which may be lock-free but uses strict ordering; a version that uses <code>fetch_add()</code> with relaxed ordering may be desirable for extra performance at the cost of not-always-up-to-date accumulation results in all threads. As of this writing this can be done by the user by providing a custom type that defines <code>+=</code> as explicitly using <code>fetch_add()</code> with relaxed ordering; but we could provide an API for this. </p>
</dd>
<dt>Namespace <a class="el" href="namespaceflow_1_1util.html">flow::util</a>  </dt>
<dd><a class="anchor" id="_todo000158"></a>Since Flow gained its first users beyond the original author, some Flow-adjacent code has been written from which Flow can benefit, including additions to <a class="el" href="namespaceflow_1_1util.html" title="Flow module containing miscellaneous general-use facilities that don&#39;t fit into any other Flow module...">flow::util</a> module. (Flow itself continued to be developed, but some features were added elsewhere for expediency; this is a reminder to factor them out into Flow for the benefit of all.) Some features to migrate here might be: conversion between boost.chrono and std.chrono types; (add more here).  </dd>
<dt>Class <a class="el" href="classflow_1_1util_1_1Basic__blob.html">flow::util::Basic_blob&lt; Allocator, S_SHARING_ALLOWED &gt;</a>  </dt>
<dd><a class="anchor" id="_todo000148"></a>Write a class template, perhaps <code>Tight_blob&lt;Allocator, bool&gt;</code>, which would be identical to <a class="el" href="classflow_1_1util_1_1Basic__blob.html" title="A hand-optimized and API-tweaked replacement for vector&lt;uint8_t&gt;, i.e., buffer of bytes inside an all...">Basic_blob</a> but forego the framing features, namely <a class="el" href="classflow_1_1util_1_1Basic__blob.html#a8c04d217b83800f818c794da17dc8e4b" title="Returns number of elements stored, namely end() - begin().">size()</a> and <a class="el" href="classflow_1_1util_1_1Basic__blob.html#a64ea173065830ef1a744458a7d05b833" title="Returns the offset between begin() and the start of the internally allocated buffer.">start()</a>, thus storing only the RAII array pointer <a class="el" href="classflow_1_1util_1_1Basic__blob.html#a4581e779dace3d64fe4098eb0a7dbeb2" title="Equivalent to begin().">data()</a> and <a class="el" href="classflow_1_1util_1_1Basic__blob.html#a719e5c04e4bdc65c31976fc385fd7f24" title="Returns the number of elements in the internally allocated buffer, which is 1 or more; or 0 if no buf...">capacity()</a>; rewrite <a class="el" href="classflow_1_1util_1_1Basic__blob.html" title="A hand-optimized and API-tweaked replacement for vector&lt;uint8_t&gt;, i.e., buffer of bytes inside an all...">Basic_blob</a> in terms of this <code>Tight_blob</code>. This simple container type has had some demand in practice, and <a class="el" href="classflow_1_1util_1_1Basic__blob.html" title="A hand-optimized and API-tweaked replacement for vector&lt;uint8_t&gt;, i.e., buffer of bytes inside an all...">Basic_blob</a> can and should be cleanly built on top of it (perhaps even as an IS-A subclass). </dd>
<dt>Namespace <a class="el" href="namespaceflow_1_1util_1_1bind__ns.html">flow::util::bind_ns</a>  </dt>
<dd><a class="anchor" id="_todo000149"></a>Investigate deeply the cause of the strange behavior with unqualified <code>bind()</code> and others described above this line in the source code.  </dd>
<dt>Member <a class="el" href="classflow_1_1util_1_1Linked__hash__map.html#a3c2ebf269951f5fe2412b6a77303b5ba">flow::util::Linked_hash_map&lt; Key, Mapped, Hash, Pred &gt;::m_value_list</a>  </dt>
<dd><a class="anchor" id="_todo000150"></a>It is probably possible to cut down on the memory cost of storing, for each element, a copy of the <code>Key</code> in <a class="el" href="classflow_1_1util_1_1Linked__hash__map.html#a3c2ebf269951f5fe2412b6a77303b5ba" title="The actual values – which, as in unordered_map&lt;K, M&gt;, are instances of Value = pair&lt;Key,...">m_value_list</a> (in addition to the mandatory one in the lookup table <a class="el" href="classflow_1_1util_1_1Linked__hash__map.html#a016890a8c1d3483041d19af810cb6867" title="Maps each Key K that is in m_value_list to an iterator into m_value_list (note the iterator points to...">m_keys_into_list_map</a>). Perhaps the key copy would be replaced by an iterator back into <a class="el" href="classflow_1_1util_1_1Linked__hash__map.html#a3c2ebf269951f5fe2412b6a77303b5ba" title="The actual values – which, as in unordered_map&lt;K, M&gt;, are instances of Value = pair&lt;Key,...">m_value_list</a>. A custom iterator class would be necessary to properly dereference this (this is non-trivial given that <code>operator*()</code> would have to return a reference to a pair which is no longer stored anywhere in this hypothetical design). Moreover, iterators exposed to the user would become invalid the same way an <code>unordered_map&lt;&gt;</code> iterator does due to seemingly unrelated changes. Finally, the memory savings would not even exist for <code>Key</code> types roughly the size of a pointer. All in all, not a slam-dunk.... </dd>
<dt>Member <a class="el" href="namespaceflow_1_1util.html#a9495b3e37e9d27bcc7767a4b6ce87ff4">flow::util::Lock_guard_non_recursive</a>  </dt>
<dd><a class="anchor" id="_todo000162"></a><a class="el" href="namespaceflow_1_1util.html#a9495b3e37e9d27bcc7767a4b6ce87ff4" title="(Deprecated given C++1x) Short-hand for advanced-capability RAII lock guard for Mutex_non_recursive m...">Lock_guard_non_recursive</a> is deprecated, now that C++1x made the more flexible <code>Lock_guard&lt;Mutex_non_recursive&gt;</code> possible. Remove it and all (outside) uses eventually.  </dd>
<dt>Member <a class="el" href="namespaceflow_1_1util.html#add56211dee74c212c19028f1ded2a785">flow::util::Lock_guard_noop_shared_non_recursive_ex</a>  </dt>
<dd><a class="anchor" id="_todo000167"></a><a class="el" href="namespaceflow_1_1util.html#add56211dee74c212c19028f1ded2a785" title="(Deprecated given C++1x) Equivalent to Lock_guard_shared_non_recursive_ex but applied to Mutex_noop_s...">Lock_guard_noop_shared_non_recursive_ex</a> is deprecated, now that C++1x made the more flexible <code>Lock_guard&lt;Mutex_noop_shared_non_recursive&gt;</code> possible. Remove it and all (outside) uses eventually.  </dd>
<dt>Member <a class="el" href="namespaceflow_1_1util.html#a9e5fae573be960434c4c9dacd48c9c11">flow::util::Lock_guard_noop_shared_non_recursive_sh</a>  </dt>
<dd><a class="anchor" id="_todo000166"></a><a class="el" href="namespaceflow_1_1util.html#a9e5fae573be960434c4c9dacd48c9c11" title="(Deprecated given C++1x) Equivalent to Lock_guard_shared_non_recursive_sh but applied to Mutex_noop_s...">Lock_guard_noop_shared_non_recursive_sh</a> is deprecated, now that C++1x made the more flexible <code>Shared_lock_guard&lt;Mutex_noop_shared_non_recursive&gt;</code> possible. Remove it and all (outside) uses eventually.  </dd>
<dt>Member <a class="el" href="namespaceflow_1_1util.html#a015b84cfa3076b8e75d16ec44a5514a3">flow::util::Lock_guard_recursive</a>  </dt>
<dd><a class="anchor" id="_todo000163"></a><a class="el" href="namespaceflow_1_1util.html#a015b84cfa3076b8e75d16ec44a5514a3" title="(Deprecated given C++1x) Short-hand for advanced-capability RAII lock guard for Mutex_recursive mutex...">Lock_guard_recursive</a> is deprecated, now that C++1x made the more flexible <code>Lock_guard&lt;Mutex_recursive&gt;</code> possible. Remove it and all (outside) uses eventually.  </dd>
<dt>Member <a class="el" href="namespaceflow_1_1util.html#a9df3cc2f3aec0c0353443b9d6c81e638">flow::util::Lock_guard_shared_non_recursive_ex</a>  </dt>
<dd><a class="anchor" id="_todo000165"></a><a class="el" href="namespaceflow_1_1util.html#a9df3cc2f3aec0c0353443b9d6c81e638" title="(Deprecated given C++1x) Short-hand for exclusive mode advanced-capability RAII lock guard for Mutex_...">Lock_guard_shared_non_recursive_ex</a> is deprecated, now that C++1x made the more flexible <code>Lock_guard&lt;Mutex_shared_non_recursive&gt;</code> possible. Remove it and all (outside) uses eventually.  </dd>
<dt>Member <a class="el" href="namespaceflow_1_1util.html#a689388b6eb7e760933d99381d65cacca">flow::util::Lock_guard_shared_non_recursive_sh</a>  </dt>
<dd><a class="anchor" id="_todo000164"></a><a class="el" href="namespaceflow_1_1util.html#a689388b6eb7e760933d99381d65cacca" title="(Deprecated given C++1x) Short-hand for shared mode advanced-capability RAII lock guard for Mutex_sha...">Lock_guard_shared_non_recursive_sh</a> is deprecated, now that C++1x made the more flexible <code>Shared_lock_guard&lt;Mutex_shared_non_recursive&gt;</code> possible. Remove it and all (outside) uses eventually.  </dd>
<dt>Member <a class="el" href="namespaceflow_1_1util.html#a62b5ba6d6a549ac20e2c055e72bef421">flow::util::Mutex_shared_non_recursive</a>  </dt>
<dd><a class="anchor" id="_todo000161"></a>Consider changing <a class="el" href="namespaceflow_1_1util.html#a62b5ba6d6a549ac20e2c055e72bef421" title="Short-hand for non-reentrant, shared-or-exclusive mutex.">util::Mutex_shared_non_recursive</a> from <code>boost::shared_mutex</code> to <code>std::shared_mutex</code>, as the former has a long-standing unresolved ticket about its impl being slow and/or outdated (as of Boost-1.80). However see also the note on <a class="el" href="namespaceflow_1_1util.html#a62b5ba6d6a549ac20e2c055e72bef421" title="Short-hand for non-reentrant, shared-or-exclusive mutex.">util::Mutex_shared_non_recursive</a> that explains why it might be best to avoid this type of mutex altogether in the first place (in most cases). </dd>
<dt>Class <a class="el" href="classflow_1_1util_1_1Rnd__gen__uniform__range.html">flow::util::Rnd_gen_uniform_range&lt; range_t &gt;</a>  </dt>
<dd><a class="anchor" id="_todo000151"></a><a class="el" href="classflow_1_1util_1_1Rnd__gen__uniform__range.html" title="Simple, non-thread-safe uniform-range random number generator.">Rnd_gen_uniform_range</a> and <a class="el" href="classflow_1_1util_1_1Rnd__gen__uniform__range__mt.html" title="Identical to Rnd_gen_uniform_range but safe for concurrent RNG given a single object.">Rnd_gen_uniform_range_mt</a>, on reflection, are very similar to <code>boost::random::random_number_generator</code>, and should be written to conform to it, so as to be usable in <code>std::random_shuffle()</code>, and also to gain its appealing extra features without losing any of the already-available simplicity. Note, however, that <code>std::random_shuffle()</code> is deprecated (and gone in C++17) in favor of <code>std::shuffle()</code>, which is available from C++11 on, and which works directly with a formal URNG such as <a class="el" href="classflow_1_1util_1_1Rnd__gen__uniform__range__base.html#a519fc83fd1c52a82618cd85f3a089668" title="The random generator engine; it is public for the reason explained in Usability section of the Rnd_ge...">Rnd_gen_uniform_range_base::Random_generator</a>. So this to-do is less about that and more about gaining those other features while being suitable for <code>random_shuffle()</code>, which some code does still use out there. </dd>
<dt>Member <a class="el" href="namespaceflow_1_1util.html#a74d4a66fb00055be4e1bd6ee98e04667">flow::util::schedule_task_from_now</a>  (<a class="el" href="classflow_1_1log_1_1Logger.html" title="Interface that the user should implement, passing the implementing Logger into logging classes (Flow&#39;...">log::Logger</a> *logger_ptr, const Fine_duration &amp;from_now, bool single_threaded, Task_engine *task_engine, Scheduled_task_handler &amp;&amp;task_body_moved)</dt>
<dd><p class="startdd"><a class="anchor" id="_todo000152"></a>We could eliminate <a class="el" href="namespaceflow_1_1util.html#a74d4a66fb00055be4e1bd6ee98e04667" title="Schedule the given function to execute in a certain amount of time: A handy wrapper around Timer (asi...">schedule_task_from_now()</a> potential limitation versus <a class="el" href="namespaceflow_1_1util.html#ae7416d64d2989051104bb396e28e15e6" title="boost.asio timer.">Timer</a> wherein each call constructs (internally) a new <a class="el" href="namespaceflow_1_1util.html#ae7416d64d2989051104bb396e28e15e6" title="boost.asio timer.">Timer</a>. A pool of <code>Timer</code>s can be internally maintained to implement this. This may or may not be worth the complexity, but if the API can remain identically simple while cleanly eliminating the one perf-related reason to choose <a class="el" href="namespaceflow_1_1util.html#ae7416d64d2989051104bb396e28e15e6" title="boost.asio timer.">Timer</a> over this simpler facility, then that is a clean win from the API user's point of view. By comparison, other possible improvements mentioned <em>complicate</em> the API which makes them less attractive.</p>
<p class="enddd"><a class="anchor" id="_todo000153"></a><a class="el" href="namespaceflow_1_1util.html#a74d4a66fb00055be4e1bd6ee98e04667" title="Schedule the given function to execute in a certain amount of time: A handy wrapper around Timer (asi...">schedule_task_from_now()</a> and surrounding API provides an easy way to schedule a thing into the future, but it is built on top of boost.asio <a class="el" href="namespaceflow_1_1util.html#ae7416d64d2989051104bb396e28e15e6" title="boost.asio timer.">util::Timer</a> directly; an intermediate wrapper class around this would be quite useful in its own right so that all boost.asio features including its perf-friendliness would be retained along with eliminating its annoyances (around canceling-but-not-really and similar). Then scheduled_task_from_now() would be internally even simpler, while a non-annoying <a class="el" href="namespaceflow_1_1util.html#ae7416d64d2989051104bb396e28e15e6" title="boost.asio timer.">util::Timer</a> would become available for more advanced use cases. echan may have such a class (in a different project) ready to adapt (called <code>Serial_task_timer</code>). I believe it internally uses integer "task ID" to distinguish between scheduled tasks issued in some chronological order, so that boost.asio firing a task after it has been canceled/pre-fired can be easily detected. </p>
</dd>
<dt>Member <a class="el" href="namespaceflow_1_1util.html#a09a032f8e1e77a3afc0b7df115a773e1">flow::util::setup_auto_cleanup</a>  (const Cleanup_func &amp;func)</dt>
<dd><a class="anchor" id="_todo000169"></a><a class="el" href="namespaceflow_1_1util.html#a09a032f8e1e77a3afc0b7df115a773e1" title="Provides a way to execute arbitrary (cleanup) code at the exit of the current block.">setup_auto_cleanup()</a> should take a function via move semantics. </dd>
<dt>Class <a class="el" href="classflow_1_1util_1_1Shared__ptr__alias__holder.html">flow::util::Shared_ptr_alias_holder&lt; Target_ptr, Const_target_ptr &gt;</a>  </dt>
<dd><p class="startdd"><a class="anchor" id="_todo000154"></a><a class="el" href="classflow_1_1util_1_1Shared__ptr__alias__holder.html" title="Convenience class template that endows the given subclass T with nested aliases Ptr and Const_ptr ali...">flow::util::Shared_ptr_alias_holder</a> <code>Const_target_ptr</code> is deprecated and shall be always left at its default value in future code; eventually remove it entirely and hard-code the default value internally.</p>
<p class="interdd"><a class="anchor" id="_todo000156"></a>Add example usage snippets in <a class="el" href="classflow_1_1util_1_1Shared__ptr__alias__holder.html" title="Convenience class template that endows the given subclass T with nested aliases Ptr and Const_ptr ali...">Shared_ptr_alias_holder</a> doc header, illustrating the pattern. </p>
<p class="enddd"><a class="anchor" id="_todo000155"></a><a class="el" href="classflow_1_1util_1_1Shared__ptr__alias__holder.html" title="Convenience class template that endows the given subclass T with nested aliases Ptr and Const_ptr ali...">flow::util::Shared_ptr_alias_holder</a>, such as it is, may well largely work for <code>unique_ptr</code> and other smart/fancy pointer types; should be generalized both in name (if possible) and capabilities (if not already the case). Could be just a matter of updating docs and renaming (deprecated-path alias may be required to avoid a breaking change); or not. Needs a bit of investigation to determine such details. (The author (ygoldfel) was conisderably less advanced when this was originally made versus the time of this writing... maybe a decade+? He meant well though.) </p>
</dd>
<dt>Class <a class="el" href="classflow_1_1util_1_1String__ostream.html">flow::util::String_ostream</a>  </dt>
<dd><a class="anchor" id="_todo000157"></a>Consider using <code>alt_sstream</code> in boost.format; it doesn't seem to have public documentation but isn't in a <code>detail</code> directory either. So that's interesting. It might have better performance than the implementation here (by being more "direct" probably). Then again it might not. </dd>
<dt>Member <a class="el" href="namespaceflow_1_1util.html#ae7416d64d2989051104bb396e28e15e6">flow::util::Timer</a>  </dt>
<dd><p class="startdd"><a class="anchor" id="_todo000159"></a>Test macOS timer fidelity.</p>
<p class="enddd"><a class="anchor" id="_todo000160"></a>Upgrade to newer Boost and keep testing timer resolutions on the above and other OS versions. Update: As of this writing we are on Boost 1.74 now. Needless to say, it is time to reassess the above, as it has been years since 1.50 &ndash; and we had been on 1.63 and 1.66 for long period of time, let alone 1.74. Update: No changes with Boost 1.75, the newest version as of this writing. </p>
</dd>
<dt>Member <a class="el" href="namespaceflow_1_1util.html#a0f681afaf67a0f917cf43a7ae893a015">flow::util::to_mbit_per_sec</a>  (N_items items_per_time, size_t bits_per_item=8)</dt>
<dd><a class="anchor" id="_todo000168"></a>boost.unit "feels" like it would do this for us in some amazingly pithy and just-as-fast way. Because Boost.  </dd>
<dt>Member <a class="el" href="log_8hpp.html#ab07c4bca2d59872736d0d9717b05aeb1">FLOW_LOG_DO_LOG</a>  (ARG_logger_ptr, ARG_component, ARG_sev, ARG_file_view, ARG_line, ARG_func_view, ARG_time_stamp, ARG_call_thread_nickname_str_moved, ARG_call_thread_id, ARG_stream_fragment)</dt>
<dd><p class="startdd"><a class="anchor" id="_todo000046"></a>Time stamp source: The current implementation uses the system clock to generate time stamps (a/k/a POSIX time), but consider optionally or mandatorily using the high-resolution clock instead (or additionally?). This has pros and cons; all or most time stamps elsewhere use the system clock also, so this allows for easy cross-referencing against other systems and logs. There's also the question of how to express an absolute time, as usually the high-resolution clock starts at system startup &ndash; not as humanly useful as a "calendar" (UTC) time, which &ndash; while useful humanly &ndash; is <em>not</em> steady/monotonic/etc. There is no reasonable conversion between <code>Fine_clock::now()</code> and a calendar time (boost.chrono docs say this unequivocally which is a confirmation). The pros include: (1) higher precision and resolution; (2) that time always flows forward and at a uniform rate without possibility of time going back or forward due to human/otherwise clock sets or rare events like daylight savings and leap seconds; or (3) to summarize, something more suited as a quick-and-dirty way to measure how long things take in the program, like an always-on, log-integrated version of <a class="el" href="classflow_1_1perf_1_1Checkpointing__timer.html" title="The central class in the perf Flow module, this efficiently times the user&#39;s operation,...">perf::Checkpointing_timer</a>. As of this writing all out-of-the-box <a class="el" href="classflow_1_1log_1_1Logger.html" title="Interface that the user should implement, passing the implementing Logger into logging classes (Flow&#39;...">log::Logger</a> implementations and <a class="el" href="classflow_1_1log_1_1Config.html" title="Class used to configure the filtering and logging behavior of Loggers; its use in your custom Loggers...">log::Config</a> allow the output of human-readable as well as sec.usec-from-Epoch time stamps. One approach might be to replace the latter <em>only</em> with the high-rez clock's time stamps, perhaps optionally, while leaving the human-readable one alone. Note: There is an important test to perform here, which is the time cost of obtaining either time stamp. E.g., the high-rez time stamp might be much slower &ndash; or maybe the opposite! To test this, (1) add the POSIX-time clock into the <a class="el" href="namespaceflow_1_1perf.html#ad61fb186191e4bb824d14601d11a2eb0" title="Clock types supported by flow::perf module facilities, perf::Checkpointing_timer in particular.">perf::Clock_type</a> <code>enum</code>, with all associated (fairly straightforward) changes in <code><a class="el" href="namespaceflow_1_1perf.html" title="Flow module containing tools for profiling and optimization.">flow::perf</a></code>; and (2) test the perf characteristics of this new clock. Certain code exists outside of Flow itself that already automatically tests all <code>Clock_type</code>s, so it would quickly give the answer. (Secondary to-do: Be less vague about where this program resides, in this comment. I, ygoldfel, have the answer at any rate and am only omitting it here for boring process reasons.) </p>
<p class="interdd"><a class="anchor" id="_todo000045"></a>Time stamp subtlety: It might not be crazy to log not just the time stamp of entry to this macro but also some indication how long it took to evaluate the rest and finally output it to the ultimate device/whatever. It could be an optional differential or something, like "timestamp+diff," with diff in microseconds or something.</p>
<p class="enddd"><a class="anchor" id="_todo000044"></a>An alternative copy-free implementation of the asynchronous <a class="el" href="log_8hpp.html#ab07c4bca2d59872736d0d9717b05aeb1" title="Lowest-level logging API accessible to the user, this is identical to FLOW_LOG_WITHOUT_CHECKING() but...">FLOW_LOG_DO_LOG()</a> work-flow is possible. The basic idea of using a thread-local non-reallocating work string is just fine in the non-<code>logs_asynchronously()</code> (i.e., synchronous) Logger flow. In the asynchronous flow, however, it involves an added message copy. Instead &ndash; as done in certain major server software author is familiar with &ndash; one could (perhaps in the async flow only) allocate a new string in each <a class="el" href="log_8hpp.html#ab07c4bca2d59872736d0d9717b05aeb1" title="Lowest-level logging API accessible to the user, this is identical to FLOW_LOG_WITHOUT_CHECKING() but...">FLOW_LOG_DO_LOG()</a> (in rare cases reallocating, even repeatedly, if more space is needed); then pass that pointer around, until it is asynchronously written out by Logger impl; then deallocate it. Thus, a copy is eliminated in the async workflow. A complicating factor is that the current system maintains a format state across separate log call sites in a given thread; this change would (if naively implemented at least) eliminate that feature &ndash; but this could be considered acceptable. (Still, do realize that, for example, in today's feature set one can set the <code>chrono</code> I/O formatting to show short unit names like <code>2ms</code> instead of <code>2 milliseconds</code>; and one need do it only once; but with this change one would need to do it in every log call site. That would be, I can attest, rather annoying. Additionally, make sure the behavior is consistent in the sync and async work-flows.) A related wrinkle is that if we add special support for <code>printf()</code>-style log call sites (a to-do in <a class="el" href="namespaceflow_1_1log.html" title="Flow module providing logging functionality.">flow::log</a> doc header as of this writing), then in that case since there is no expectation of such format statefulness in the first place, in that flow that particular concern isn't a concern. (Sub-to-do: If one did this, an extra-optimized further idea is to avoid the repeated allocs/deallocs by maintaining a pool of already-allocated buffers to be reused opportunistically.) Bottom line: I claim the existing thing is pretty good; the extra copy is IMO unlikely to affect real performance, because (1) it's only one copy in the context of quite a bit of similar copying and other ops going on while writing out the string; and (2) if the message is so verbose as to affect performance <em>at all</em>, then it will probably affect it regardless of the extra copy (in other words, its verbosity must be increased, or the filter verbosity must be decreased &ndash; avoiding this exta internal copy feels in my [ygoldfel] personal opinion like rearranging deck chairs on the Titanic). So, this to-do should probably be to-done at some point, but it doesn't feel urgent. And since there are quite a few subtleties involved, as shown above, it's natural to resist doing it until truly necessary. </p>
</dd>
<dt>Member <a class="el" href="log_8hpp.html#a626c7dc4d3b4dc0b32a8aac8624d66bc">FLOW_LOG_WARNING</a>  (ARG_stream_fragment)</dt>
<dd><a class="anchor" id="_todo000042"></a>We can avoid using macros for this and similar APIs by requiring the user to use commas instead of the usual <code>&lt;&lt;</code> and by implementing this as a variadic function template (C++11 feature). </dd>
<dt>Member <a class="el" href="log_8hpp.html#a692bbadd8b5c3e9cef7a0bbea1915e66">FLOW_LOG_WITH_CHECKING</a>  (ARG_sev, ARG_stream_fragment)</dt>
<dd><a class="anchor" id="_todo000043"></a>In <a class="el" href="log_8hpp.html#a692bbadd8b5c3e9cef7a0bbea1915e66" title="Logs a message of the specified severity into flow::log::Logger *get_logger() with flow::log::Compone...">FLOW_LOG_WITH_CHECKING()</a>, save <code>get_logger()</code> and <code>get_log_component()</code> return values in such a way as to be reused by the <a class="el" href="log_8hpp.html#a5daa2b6d16edea74bb8bddc75f7fb801" title="Identical to FLOW_LOG_WITH_CHECKING() but foregoes the filter (Logger::should_log()) check.">FLOW_LOG_WITHOUT_CHECKING()</a> invoked by the former macro if <code>should_log() == true</code>. As it stands, they are called again inside the latter macro. In the most-common case, wherein Log_context is used for those two expressions, this should get inline-optimized to be maximally fast anyway. With <code>FLOW_LOG_SET_*()</code>, though, it might be a bit slower than that. Technically, one can make their own <code>get_logger</code> and <code>get_log_component</code> identifiers that might do something slower still &ndash; though it's unlikely (and as of this writing unprecedented). So I would not call this pressing, but on the other hand... just do it! The implementation code will be somewhat hairier though.  </dd>
<dt>Member <a class="el" href="util__fwd_8hpp.html#a0cc4c82ac54e53b7a8a2431b66ff30df">FLOW_UTIL_WHERE_AM_I_STR</a>  ()</dt>
<dd><a class="anchor" id="_todo000170"></a>See if <a class="el" href="util__fwd_8hpp.html#a0cc4c82ac54e53b7a8a2431b66ff30df" title="Same as FLOW_UTIL_WHERE_AM_I() but evaluates to an std::string.">FLOW_UTIL_WHERE_AM_I_STR()</a> can be coaxed into a compile-time expression after all. It is used in quite a lot of frequently executed code, namely at the top of most Flow (and Flow-like) APIs that can emit errors. See notes inside <a class="el" href="namespaceflow_1_1util.html#a3a33fa903d51cbe124781544eaf0c4bc" title="Helper for FLOW_UTIL_WHERE_AM_I(), etc., that, given values for source code file name,...">flow::util::get_where_am_i_str()</a> on this as of this writing. </dd>
</dl>
</div></div><!-- contents -->
</div><!-- PageDoc -->
<!-- start footer part -->
<hr class="footer"/><address class="footer"><small>
Generated on Fri Mar 28 2025 22:55:35 for Flow by&#160;<a href="https://www.doxygen.org/index.html"><img class="footer" src="doxygen.svg" width="104" height="31" alt="doxygen"/></a> 1.9.4
</small></address>
</body>
</html>
